Game performance loss after few games

Hi, I am running a latest gen computer with RTX570ti and having a strange issue. The game runs normal and after few games it drops performance about 20-40%. This is immediatly fixed by restarting the game. I would be more than happy to wait longer for a map to load if you only start a match with a clean 'buffers'. This was not happening in beta for sure.