Cannot start a rivals match

After the cutscene of opponent formation when the game should begin, the game says “Lost connection to your opponent…”

This happens to me only on rivals as i completed icon swap objectives without any issues. As i went back to rivals just for fun, cannot start a game. I tried restarting my ps5, the game and wifi connection but it still persist. The day before as it did this for the first time, a game restart would let play just a single rivals match before it starts happening again.