Frame drops despite specs being more than capable of 120FPS

After a recent update, on Xbox Series X, the game's frame rate will randomly drop to around 5-15 fps, as also shown by the performance display on the top right.

This drastic fps drop will occur after 20/30 minutes of the game running, and can happen everywhere from main menu to firing range or in battle.

The only way to get the 120 fps back is restarting the game.

The console is not overheating and runs any other game just fine. I know other people are experiencing this, so the problem is not my device.