I don't usually feel bad about deleting people. It's real easy to make new friends. I do hesitate more with friends I've had for a while. I have one friend who was really good Halloween-Valentine's, but hasn't had anything to tap for a few days. People take vacations, drop their device and break it, have connectivity issues, have real lives, so if they've been a good friend overall and it's not been too terribly long, I keep them. What's 3 FP for giving someone who is good most of the time? So that user gets to stay for a while longer.
If I get a random friend request and that person is obviously inactive quickly, they get booted, regardless of what their last login says.
I didn't delete farmers before just for farming's sake, but I did try to get rid of the ones who crashed my game all the time or the landing zone dumped me in the middle of their farm.
I try not to delete people for not visiting me, but during Christmas, some of them got booted too.