Thanks everyone for helping test this issue. I'm confident now, that with the two methods that seem to work to get rid of the stacks, the game can be playable as long as you don't keep stacks of items in your inventory (special stacks with the stack icons)
Please see my blog post for more details: https://sims4.crinrict.com/eng/2024/03/stacked-inventory-items-crash-the-game/
Let me know if you have more issues or if the methods don't work for you.
Please use this thread if you have issues with stacks in the household inventory: https://answers.ea.com/t5/Bug-Reports/Game-crashing-when-selling-moving-from-build-buy-mode/td-p/13525292
MOD FIX
So NeedCoffee4That created a small mod for this issue: https://www.simfileshare.net/download/4513178/
It uses the proposed suggestion here: https://answers.ea.com/t5/Bug-Reports/WORKAROUND-Stacked-items-in-inventory-household-inventory-cause/m-p/13527250#M272080
What is does is getting rid of the multiple different items stacks. It only stacks brooms etc that are exactly the same broom and not all brooms together.
This is a simple band-aid and not a proper fix but it stopped the crashing.
The mod isn't thoroughly tested so please use at your own risk. I did some testing with the brooms in the two saves I have and didn't see any issues when using the brooms.
Testers are still needed for light sabers, kyber crystals and hilts. Let me know if those work too.