Personally, I'm not so sure what the fuss is about. I get it that not everyone has the same end goal look for their town- some people want it to look as close as possible to the one in the show, others might just place things without a care, and others will organize things to make things easier to tap and leave. Personally, my end goal is the first while still being efficient to clear, but it takes a lot of time to reorganize things, and it gets a lot worse due to new content and new land being released all the time. So while I understand it can be frustrating to spend a lot of time on your own town while other people's towns clearly haven't had the same effort spent on them, I also think we should recognize that not everyone has or wants to spend the time arranging all of their buildings or decorations just so. Different people play the game differently and I don't that's any reason to judge them.
So maybe some of your neighbors can be annoying to tap, or take too long to load, or may even just be plain ugly. Why not just delete them and find someone else to add? They're not actually your friends, so I don't think anyone should feel bad about deleting them.