Cant unset keybinding

I want to bind a mouse button to an action. However it was already bound. No problem. I'll just undbind it...

There is no way to unbind specific actions.