Can't change lot from rental back to residential lot

I built a rental property and after I was done I save it before bulldozing it. Then I wanted to build a residential house on the property, When I wanted to change lot back to residential it saying your Sim is currently in this unit and the property type cannot be changed. I've tried moving Sims in and then evicting them. I've tried without mods but nothing. The property is empty there's nothing there no building, no Sims. It's essentially just an empty rental lot now. How can I fix this?