Motion Controls Not Working Properly (BFN Switch)

Does anyone else playing BFN experience this weird "motion drag" when aiming with motion controls? 90% of the time when I'm playing, my camera will try to drag itself into some direction and it makes playing the game a pain. It's not stick drift, I've checked. This ONLY happens on Battle for Neighborville, and I've tried out both JoyCons and a Pro Controller and it happens to both only when I use motion controls. Is there a way to fix this or is the game simply bugged? I want to play the game but this is so frustrating to deal with.