I can open map with mouse4 but cant close map with mouse4

I bound "Big Map" to mouse4 (back button on mouse). it works to open map, but once map is open i cant close it with mouse4. Then i try pressing "esc" but that only opens the menu. So now i find myself in the middle of a fight with the minimap and the menu open, crazy design choice imo.

Anyway, please fix