You can remove occult states with MasterController (Cheats module required). I don't know how to cap the mermaid population. NRaas StoryProgression has a Town Occult Ratio setting under Options: Sims (going by the interactions page). Default is 50 for each occult. However I'm not sure what that controls. The interactions page suggests that setting only controls newborns and SP's generation of immigrants.
Aren't there TCE options on the...mailbox for occult populations? I never used that and don't know if mermaid population is included.