Playing College Football cross play

Hi - I have an xbox and am trying to crossplay college football with a friend on a playstation.  When I set it up and invite him, once he accepts, the controller moves from the right (my side) to the left.  We want to play against each other as different teams so not sure why controller keeps moving to left.  Please help.  I am subscribed to xbox core.