If I delete all residents/townies will it break my game?

The only thing you need to do is make sure you have 1 household marked as played, so you can't delete every household, you have to have just 1. Having no households marked as played makes the save feature wonky, It will not work properly and can overwrite other files.

Other than that, you can delete all the households, just make sure you evict them from any lots first. Deleting households who are assigned to lots can cause the lots to loose all the furnishings.