If you periodically prune your neighbors...

I only accept players at max level. After each level up, if a player still haven't upgraded within a week, I'd delete the player from my neighborhood, to make room for other players in my waiting list.

I don't have time to track who visited me, and I don't bother if they are house farmers. What matter most is that they must be active daily players like myself, so that I can rely on them when big event comes.