FPS dropping as soon as I am in a game!

Every time I play a game, the first couple of mins is fine but then suddenly the FPS drops drastically and the game becomes unplayable. My PC is fully capable of running this game (I will put specs below). I have even put the graphics setting on LOW. Is there anything else I can do to sort this out. 

My PC Specs:

Processor    AMD Ryzen 7 3800X 8-Core Processor                3.90 GHz
Installed RAM    16.0 GB
Storage    1.82 TB HDD ST2000DM008-2FR102, 932 GB SSD Samsung SSD 980 1TB
Graphics Card    AMD Radeon RX 6600 (8 GB)

Please help if you can. Thanks!