I have a few friends who are virtually inactive, with very low counts and hardly anything to tap when visiting, but as I've crafted this ideal neighbor list over the years, the people who aren't playing as much must have something going on and since I don't have trouble meeting goals, I can wait longer for them.
A year ago, I would have said 3 days of nothing to tap during an event would do it, or 2 weeks when an event isn't active.
Ultimately, Mike, it's up to you. Delete those you want to delete and PM some forum regulars that seem to have the same passion as yourself for the game, and see if they have a spot open.