Premium friends causing trouble for players with DLC but not premium?

I have the They Shall Not Pass DLC but not the premium pass (XBOX ONE). Since the recent update, it seems that I can only play with friends who have the premium pass or DLC. If I try to matchmake on non-DLC maps it generates an error every time. The only way to play with friends who don't have DLC or premium is for them to join a game then I can join theirs once it's loaded, but even that is hit and miss. 

Is there a way around this other than buying the premium pass... or a way to disable this premium friends setting? Only a few of my friends have DLC content...