I'm not sure why your computer thinks RTSS isn't compatible with your OS. Are you running Windows 10? That's what I have, last updated in July, and RTSS version 7.1.0 works just fine. I did have trouble installing it initially, though; I went back and checked the messages from when igazor was helping me through the Bootcamp process, and I fixed the issue by trashing the installer and redownloading through Chrome instead of Firefox. On the other hand, at that moment several features had decided to all not work at the same time, so there may have been some underlying problem of which I'm not aware. (By the way, igazor has Windows 7 and RTSS 6.6, for what it's worth.)
I don't use MSI, but both it and RTSS should work the same in fullscreen and windowed modes. RTSS at least doesn't distinguish between the two—the profile you set up for TS3 applies in both. (Again, the app does need to be running while you play.)
It just occurred to me that you might have the Windows 10 Creator's Update. A year ago, it was giving TS3 players fits, although the issues seem to have been worked out now. But it could conceivably be causing your issues. You might experiment with turning off its effects, to see if it makes any difference:
https://forums.thesims.com/en_US/discussion/comment/15902876#Comment_15902876