Losing fame, but still having negative effect of fame quirk?

I had a 1 star celebrity who got the "vanity" fame quirk, so he'd get tense if he didn't admire himself in a mirror for too long. Then his fame went down, so he "lost" the quirk, but he still gets tha...