It's your game. That means you choose the design of your own town.
It also means you choose your neighborhood. One of the only times in life you can select your neighbors. So do it!!
If you want a neighborhood full of well designed towns, that's totally your call and no reason to feel guilty.
I personally don't care too much because sometimes events require neighbor visits with a lot to tap, and sloppy noobs or house farmers often have items left when other neighbors don't. So don't shoot yourself in the foot eliminating quicker visit neighbors.
Guilt should only arise when you criticize your neighbor's creativity unfairly. We all work our own town. But otherwise, you are free to play the game how it pleases you most!
Happy tapping!