My friend lags when I invite him, and I do when he invites me

Me and my friend (we live one hour away from eachother, so no we are not on different servers) are trying to play against eachother in Ultimate team friendlies. However, when I invite him, his ping is very bad and has an insane delay (The red internet icon thing appears too),  but my ping is just like normal. When he invites me, its the exact opposite: I lag the same way he did when I invited him, and now his ping is back to normal. We don't know what the reason behind this is and we can't find a way to fix it, Im pretty sure we lag when we get on coop with eachother too however that was one game we played in coop so I cant fully be sure if thats the case in coop too. We are both on PCs incase that has something to do with it,

Please if anyone knows how to fix this issue let me know, thank you!