@eclecticallyem You might need to do a bit of experimenting to figure out what the problem is. Remove your Sims 4 folder from Documents\Electronic Arts again, and load the game to create a new folder. Start a new save (no need to spend much time on it; this is just for testing), place the sim in a house somewhere, save, quit. and restart your computer. Then, without adding anything else to the new Sims 4 folder, launch the game again, just to make sure it still loads as quickly as it did before. If loading is slow again, stop here and let me know.
Otherwise, you can try adding your existing content to the new folder, a little at a time. It's best to use copy and paste, not drag and drop—that way, you'll have the original files and folders as backup, and you can freely experiment on the copies. Start by copying the contents of Tray from your old folder to your new one, and see whether ti makes a difference. Then you can add the contents of Mods, and test again. If there's anything else besides your saves, add that next; leave your old saves for last.
Let me know whether any files cause the game to load slowly again. If some do, you can remove the entire batch you just added, test to make sure the game loads quickly again, and then add half of the batch that caused the issue. If the slow loading is back, split that half in half and test again. If half the batch doesn't affect load times, then test half of the other half. This way, you can narrow down the source of the issue fairly quickly, even with a large number of files.