I agree this should be something already in the game.
But for those using mods and not knowing about this:
LittleMsSam has a mod that let you fix that.
It's called "Gender & More Custom Lot trait"
https://littlemssam.tumblr.com/post/175412863403/gender-more-custom-lot-trait-this-mod-adds-a
Give this lot trait to any lot you want. When you visit the lot in game mode, click on the door and disable so many groups you want.
What's really cool about this mod is that you can choose from: celebrities, paparazzi and fans. Which means you can disable them all, but you can also create a lot where celebrities do come, but no fans or paparazzi.
I play with this mod for two years now I think. And I have never had a celebrity on a lot with this mod and these settings.
