It depends upon the size of the inventory. If multiple Sims in a household have lots of stuff in their inventory, it does slow down the game. If you have a lot with a lot of clutter, it has the same effect. Your computer CPU has to account for all of it, and if it gets rendered on screen, it also involves your GPU.
One good habit to develop in game is to regularly inspect Sim's inventory to see what has shown up in there. Sims accumulate a lot of random stuff, and it doesn't hurt to clean it out. Most things in inventory can be sold, and that helps with household income.
Household inventory, located in Build/Buy Mode, is another place that things accumulate, and some of those items can fetch a pretty penny if you sell them.
If you have Eco Lifestyle, recycling items in your inventory is a great way to get resources for fabrication.
All of these things are ways to improve game performance and increase household income at the same time.