Insane Packet Loss When not hosting

My fiancee and I play on Xbox, myself on Xbox series X and her on Xbox series S. Ever since the server updates, anytime we play, the person not hosting suffers crazy packet loss to the point where you play for one second and Rubberband backwards for 3. We both play in the same house on the same Wifi, both Xbox have ethernet plugged in, and we both have run network test with 0% packet loss showing on either. We also play other games like Marvel Rivals and Fortnite with no issues, only seems to be this game. Any help would be appreciated, we miss playing everyday lol