Display delay when playing online

At first I thought it was gamepad input delay but as I was playing online with a friend and with "3, 2, 1, crossing now", it took 1-2 seconds on my part for a cross to be kicked while on his end it was on "now".

This happens randomly, I can restart my PC and the problem can be gone or not. I can play 1 - 3 games than 4th game will be a problem or the problem can be for all games. It is unplayable in such conditions. Connection score is 96.