How can we get more than 4 friends on the same server?

You cannot form a party of more than 4 people, there is no option to join the server a party is already playing on, nor is there any way to send an invite to others outside your 4 man party to join a server. I cannot understand why we have a 128 player server but cannot have more than 4 friends playing on it.