How inactive does a neighbor have to be before you delete them?

It depends... Due to some problems like the harp which left players outside of their towns for weeks you should give them at least a week or two before deleting them. I do it if they havent logged in ...