Both Inspector and RTSS will work in windowed mode and thus in Windowed Borderless Gaming, just in different ways. Inspector imposes its settings changes on the card in the same way as the Nvidia Control Panel only it reveals more options that the Control Panel does not, RTSS uses its own mechanism and in fact must be running the entire time that the game is in order to have any impact at all.
It is vital to cap those fps rates in windowed mode the same as in full screen one way or another if your intention is to play in windowed or both modes. The runaway frame rates will affect RAM usage but will also lead to crashes for other reasons. The game should not be causing your card to generate frame rates that are over the refresh rate of your monitor as your monitor will never be able to interpret those and they are the equivalent of digital noise.
There's nothing wrong with trying MSI Afterburner either, but my sense is that it does the same thing that RTSS does. Or rather, the other way around as RTSS was meant to be a substitute for MSI for those who don't need all of its other features. That might be the early history though, the two may have diverged from each other since then.
Yes, all of these tools work on Nvidia cards. I don't even have an Nvidia card, mine is also AMD, so am relying on what others report when it comes to the tools and their interactions with the cards.