Trying to play with friend but always get "server full" error

I tried restarting the game on my end and his end, switching between joining/inviting but it always states that the server is full.