It Takes Two - "The other player has left the session"

I'm playing It Takes Two on my Steam deck, and we've made it almost to the end of the game. But we keep crashing every so often, sometimes after an hour, sometimes after 10 minutes its random and always gives the message the other player has left the session. This issue only began in Rose's room, not before, and hasn't subsided.

What gives? is it from lag we do have very high latency sometimes, and is usually before a crash.