Input lag when queued with friends?

I usually play solo, but the few times recently I have queued with friends, I will randomly get input lag, specifically when aiming. I play on Xbox, and whenever this happens when I go to move the right stick, there is a slight delay in my input and what I see on the screen. Its a wireless controller, and the only device that is connected to my console is my wireless headset. Again this only happens when I'm queued with someone, and restarting my game fixes it. Has anyone else run into this issue?