How to remove the spellcaster trait from townies?

I've found that if you make a bunch of spellcaster sims (or download them from the gallery) and put them all in individual households (they can be not in world), the game will draw from that pool before turning random sims already in the game into spellcasters. Depending on how long you plan to stay in the magical realm, you'll need at least 12, probably closer to 16 to 24.