I see how many actions I have left at the end of the day. As long as it's below 50 or 40 or so, I figure I'm not missing out on too many FPs. Usually I'm around 30 actions left over which means 10% of my friends had nothing to tap. If they haven't been active for weeks, I will drop them, though.
The main problem with inactive friends isn't just that they have nothing to tap, it's that I waste a minute or two scrolling through the whole darn town. It really slows down the visits.