Right mouse button zoom glitch

When I hold the right mouse button to zoom there is a glitch where, especially when aiming at an enemy, it won't stay zoomed forcing me to hip fire. 

Typically it only happens when looking at an enemy but in the attached clip you can see it going bonkers. The entire clip I have the right mouse button held down to zoom. 

Not sure if it is tied to latency but my ping is usually around 50-60ms rarely spiking to around 75ms. I've tried to glance at my ping when the glitch is happening but I haven't been able to correlate the two. 

Not sure how else to describe the problem but it's pretty game breaking for me and obviously incredibly frustrating. I would literally have to switch to a hip fire weapon which doesn't go well in the wide open maps.