Romantic Relationships decay/become negative fast with no interaction

This is very annoying, I have a legacy family and sometimes have to switch to make sure they are aging somewhat realistically, and each time I do so, the romantic relationship with my current sims falls to 0. I literally just went to another family, did some aging up and went back. That wasn't even a day in game - it was like 8 in game hours. And they were already just friends.