This is anecdotal because I always look for the vandalize buildings and since I have to move around people's Springfield's so much I've noticed...
I've noticed that sometimes there are no buildings to click on neighbors buildings except fast refreshing brown buildings and sometimes every building is available.
I think that if I collect rent in my town and immediately someone visits, they won't see anything they can click except maybe the fastest growing (brown buildings)
If this is correct, if you only see neighbors hitting your brown buildings, you may be harvesting too fast and on your game too much. :?
EDIT: you can never be on this game too much!