Performance worsening after several games in a row since around last patch

I will start playing, and the game will be fine, 80% GPU usage at 60fps, second game, I don't notice anything. Third the game will start dropping frames, with the GPU at 99% usage and frames dropping to 50 something, and if I the game ends up being long, to the 40 something. It's not temperature throttling, I'm monitoring them. It was either the update, or the latest Nvidia drivers, 595.97, since I updated them the same day the game updated. Before that I could play as many games in a row without issue.