Remote control

The last few times my remote has gotten unplugged I can no longer play on the remote. I have to shut the whole game down and restart for it to work again. What am I doing wrong