Can you become unfamous?

I haven't started to get famous in the game yet but was wondering if once you are on top of fame if you can lose it too and have to keep working at keeping the fame.