Can I stop my world from auto-populating?

I've been trying to build a world where all of the Sims in it are ones I've made myself, but the game keeps moving in new families and messing it up! I've tried to turn off "Fill empty homes" but now ...