Non-host in squad kicked out when me and brother are playing under same WiFi

We are playing on Xbox Series X. I recently had two routers that caused double NAT issues, but connecting to router 1 and my brother connecting to router 2 allowed us to play. 

After getting rid of router 1, we can no longer play together as I believe it’s because we’re under the same connection now. The non-host is unable to load into the match upon loading in every time. We have UPnP on our router, which is a Netgear Nighthawk.

Any fix or assistance would be greatly appreciated, this has been a problem for months now but is now unavoidable under my circumstances.