Sims are only allowed to be one type of occult at a time. That shouldn't stop a Spellcaster, Vampire, or Werewolf from becoming a merSim, as long as the previous occult state has been removed. I just tested this and had no trouble switching from Spellcaster to MerSim. Here are the steps I followed.
1. Travel to the Realm of Magic.
2. Ask for the Rite of Dissolution.
3. Say Ok.
4. Get a piece of Mermadic Kelp. (I bought one from the Rewards Store)
5. Eat the Mermadic Kelp.
6. Go somewhere that has swimmable water. (I chose Gibbs Hill in Britechester)
7. Go swimming.
Perhaps you missed the last step?
Yea, I did all of that...I even ate the kelp while standing in water
My best guess, then, is that you didn't swim out deep enough.
