Xbox Series X Controller Disconnects

Game runs on Xbox series x with ethernet cable. Offline modes such as UT Squad Battles are completely fine. 
When playing Rivals, Xbox controller randomly disconnects and connects again. Meanwhile for that 2-3 seconds i cant move, pass or shoot. 
I tried with 2 different controllers. 
Offline modes and other games works fine. 
Controllers and console are updated. 
I know there is a similar issue for pc players. 
Anyone have same problem as me on Xbox / ps5?