Ok, started over according to plan. Meaning I donwloaded about 100 sims more or less randomly and left them "not in world".
Obersvations (i run MC Commander):
1. Game stutters like heck the first two or so game days because both SIms and MC Commander is trying to assign Sims to work places as well as clubs etc.
2. It still sprouted a number of random sims. My solution (unless they actually looked okay, in which case I just improved their clothing) was to edit household, download a bunch of sims into that household, delete the original and rename the household to something fitting.
3. After a week everybody that could fit anywhere was randomly moved in by MC Commander. I am standing by checking every sim morning if the game has generated new townies and if it has add 2-8 of freshly downloaded ones instead.