FPV changes directions when I ADS.

In some instances I have found that when I ADS, my camera and FPV changes to a completely different direction than I was aiming at. It's happened multiple times and I have tried different button presses to try and trigger it manually, but I can't figure it out.