I have had this issue too, mostly with people from within my country (I live in Norway), and we pretty much all have the same ISP. I have a VERY stable network too, with every port open, so normally, this should never be a problem locally. It is 100% server side.
However, when we connect to a foreign host, there was no issues at all. There has to be something wrong with the netcode altogether, that it sometimes blocks players on the same ISP from properly connecting to eachother.
Either way, it is weird that the problem mostly occur with friends who live closer to eachother.