You generally need to have some sort of mod that changes or limits townie generation for the game to not generate random townies since the game is coded to generate sims for filling all sorts of roles, from townies to people tending the bars etc.
In the past I used mods like npc control by Paulson or no random townies by Zer0 or all the modules from townieoverhaul by Bienchen (Bienchen's mods is probably what I would recommend)
These days I use mostly some custom settings in MC Command Center (population settings set to 0 for all ages (Population_Percent)) and just one module from bienchen called townieoverhaul nonewbarflys because the game generates a crazy amount of townies every time you visit the bar.