Sims 4 - game freezes when I exit CAS or Build Mode

Whenever I exit CAS or Build Mode in TS4 my game freezes up. It has the red outline like when you pause the game, but the pause UI is gone and I can't click on anything or make anything work.

The only way to play is to exit the game every time it happens and then reload it, it's super annoying and I can't find answers, please help!