Players being invisible

After playing for a little while I have noticed that players end up not being visible, both friends in a party and just being online in a server with randoms. This is annoying as the only way to fix this is to restart the game or just spectate the player you want to see.