No visuals in headset in VR mode

When I start the game, it asks me whether I want to enable VR Mode, once I hit confirm, it continues but there are no visuals in the headset. The settings menu has the VR options, the rendering in the stages looks like the FOV-eated rendering on screen, but nothing inside the HMD. I have tried this with a Pimax Crystal (not officially supported), HP Reverb G2 (officially supported) and Quest 2 (officially supported), both in SteamVR Mode as well as OpenXR Mode. I have not tried it with Oculus' VR standalone VR runtime however.