Input will randomly switch mid-match from MKB to Xbox Controller

This has happened a few times when I have left my controller plugged into my PC. The MKB will be the input for one or several games, and then it will suddenly switch to Xbox controller (I don't touch it and nothing is touching the controls) for the look only. My movement keybinds are still MKB at that point, but my look is locked to only respond to my Xbox controller. The only way to get my look to go back to my mouse is to unplug my xbox controller.