Either you deleted them or they deleted you. They will show up in the log as Anonymous after that.
So wait, if you unfriend someone they can still go to your town? Do they have to unfriend you too in order to not see your town anymore? I am confused.
It looks like you're new here. If you want to get involved, click one of these buttons!