can you change sims from the gallery to occult sims?

You can’t make sims into aliens or turn aliens into anything else. But you’re free to turn sims into vampires, mermaids, or spellcasters the usual ways.

It's possible to make an NPC sim an occult without playing it, but it requires cheats and knowing the Sim ID or MCCC (which is easier)
Gallery Sims will need to be installed in a house to be official NPC's (I think - at least they can't say in the gallery) but you can then save them back to your library.

Actually, if you don't mind cheats you can easily turn any sim into any occult type, including aliens. First, the MCCC mod has an option to easily turn any sim into any occult type except for a spell caster. With cheats, you can also easily turn any living sim into a ghost without going through the whole death and dying part.

If your interested in more information about those cheats, they can be found in the spoiler.