Friends entering town = getting kicked from game?

Does anyone know if visitors are causing all the of the drops from your town? I know I'm not the only person getting kicked out every few minutes.

I'm about to clear out my friends list, but don't want to if it does nothing to help.


Also, an unrelated question - anyone know how to decline friend invites on the origin software for pc?