Can't drive with XBox Wireless controller if Keyboard/mouse attached

So if I can't play this game at all! I'm trying to play with the XBox One Wireless controller for Windows, but there seems to be a conflict with my keyboard. All the controller buttons are assigned to their defaults and work in menus (except B randomly), but all the menus show keyboard controls unless I'm holding a button down on the controller. If I exit the garage the car just wheel spins when I press any button on the controller and it won't let me drive with the keyboard either. I've tried unbinding the buttons but nothing seems to work. I found another thread where someone had the same problem but their solution didn't apply to me (it was to disable an infrared device in device manager, I don't have any).

Any suggestions? I don't exactly want to unplug my keyboard and mouse every time I play. Is this issue also present in NFS Payback for anyone?