Would randomly join a random party when I tried to join my friend's

Like the titles says, whenever I tried to join my friend's party from another server, I would randomly join a party within my server's world. This only occurs around 70% of the time.