Issue with VRAM

First game I play is always amazing, getting 120+ fps and having a fun time. But as I play more games it seems that the fps only drops game by game. If I look at my vram usage it constantly climbs until I am forced to restart the game to play again. I have looked and looked for a fix for this and it seems that the new update has only worsened it.