Randoms joining party/teleporting despite 'Friends Only' in party settings

Had this happen a few times already though today, I was unable to leave the party or kick the person. Tried to block them and was still stuck in the party with them. All party settings set to Friends Only, including spectate/teleport.