Inventory Toggle keybind doesn't toggle

Toggling the inventory open or closed is defaulted to TAB, which *does* toggle the inventory.
When setting another key to the that bind, it only opens the inventory but doesnt close it, forcing me to hit TAB instead or click close in the lower left.