The camera starts scrolling without touching mouse

This has happened twice to me now, seemingly randomly the camera starts to pan upward until it reaches the edge of the map. It keeps moving forward and you cannot move your camera back or left, cente...