Update: The issue has progressively become worse with frequency and intensity ONLY with the new controller (that came with the Series X) which I had thought I could rule out of not being bad. With this new controller, I have to set the dead zone to 60 for this issue to go away, which is totally unplayable. My older controller which I know does have a slight stick drift I can get away with the dead zone set to 18.
In conclusion, I didn't think it was likely, but it turns out both of my controllers (including a new controller that came with my new Series X) have stick drift. Adjusting the dead zone helps to a certain point until it becomes unplayable, meaning I will need to get a replacement controller.
