Ideas
Yeah the frame limiter seems to be the main culprit behind this. With it off I'm hitting around 160 FPS in game. When I turn the limiter on and set it to like 150 or 160, FG will immediately take over with my FPS being only around 130 to 120? Which seems odd. Its like FG says "ok, don't worry we can hit 160 for you, CPU/GPU, you take a break."
If however I set the frame limiter to above my normal frame average, so around say 200. I'll never experience FG limiting my actual FPS. Something odd is going on in this game when FPS is limited, It seems if the FPS cap is set just close enough to hit where you would with raw render FG gets turned on in the background and your GPU/CPU scale back some at the same time, the FG lets you hit the FPS cap you set, albeit using FG instead of raw rendered frames.
TLDR: FPS limiter seems to be the main culprit. Setting a limit turns FG on and instead of letting your GPU/CPU do the work to hit that frame rate, it takes load off of them and uses generated frames to hit the limit instead.
I turned off the ingame and Nvidia control panel cap limiter, and instead used the Afterburner/RTSS async cap limiter. I haven't seen the FG indicator come back on since.
What's also weird is that it significantly improved the 1% and 0.1% lows. The avg FPS only went up by a couple from 149.8 to 152.2, but the 1% low improved by 8 (up to 112) and the 0.1% low improved by 12.4 (to 107.2).
It also really smoothed out the frametimes, almost entirely removing 8ms+ frame to frame differences, and changing the 2ms frame to frame differences from being only 16% all the way to 28% (the remaining frame to frame time being 72% for 2.1ms to 4ms.