Let's try to get to the bottom of the lag issue.
Which console are you on?
I am assuming you are in GnG?
I am also assuming you tried clearing the cache etc? (Xbox One instructions below)
Power cycle your console. Here’s how:
On the console, press and hold the Xbox button until the light behind it turns off (about 5 seconds), which means the console is off.
After the console has shut down, unplug the power cord and wait 30 seconds.
Plug the console back in and turn it on using the Xbox button. Power cycling will reset the network card and clear the cache.