Guy sitting in my club now can't get into 3s

A guy I was playing with got mad thaty i removed him for trash talking my play style he started trolling and now everytime i try to join my club on 3s it says invalid game session and says my xbox comminucation settings are preventing me from joining the party when i have him blocked then he messaged me saying the only way to not permently glitch my club was buy ejoining in 5 minutes and i don't want to lose all my club progress but i also don't only want to play 6s