I've had long loading times occasionally and in the past when I've tried emptying out inventories it never seemed to make a difference. I wouldn't worry too much about items in inventories with respect to loading times. I've found that reducing the number of household members and reducing the numbers of objects on the lot (not in inventory) can help. In long played games though, sometimes the only thing that really helps is relocating the household to a new world, which I tend to do via Traveler.
I don't clear caches every game session. I notice it takes longer to fully load lots if I do, particularly with textures. I find it better to leave them be (particularly simCompositorCache and compositorCache) unless I'm trying to solve some other issue or adding/removing CC.
I've had one game which would quickly grow to 20+ min load times. The only thing that worked to reduce that was to use MC to reset the whole town (including the active lot) once a sim-week. I have no idea why an MC reset was so helpful and have only had that help in one particular game. The loading time after resetting went back to a more reasonable 7-10 min, IIRC.
Good luck.