VR is broken

Whenever I start the game in VR mode, everything looks fine, until I get to actual gameplay (garage/driving). The view is broken, each eye looking in some random direction. Resetting VR position does nothing. I have also tried many other options, tweaking settings, closest I got to "fixing" it was overriding scale to 50%, but even in that case, it is still slightly headachy, and just looks overall horrible.
The problem is on PC, using a PS VR2 connected through the official PS VR adapter, with an AMD RX 9070 XT through DP 1.4. Other games work perfectly fine with no tweaking.