Fps drop from 100fps to less than 30fps

Generally my game runs very well all on ultra, at around 100fps, but sometimes when I play the game a little, for example, after replays, or end of missions (I believe when it runs a replay) my fps drops bellow 30fps even after the replay is over.

Could you please solve this problem?