Very frustrating. I can't visit other towns anymore without being booted off the server.

You may have a corrupted neighbor database.

• Write down the names of all your neighbors.
  
• Unfriend them.
  
• Friend them back one at a time.
  

Really though you don't need to visit neighbors that often once you've hit the peak. I had stopped visiting at all and now I'm paring down because it's just too much trouble.