How to upgrade follower's gear?

This is probably obvious, but I've only found upgrade tables in Haven, but you don't have access to your followers there? Do you really have to take their weapons and armor away, travel to Haven, apply mods, travel elsewhere, and give them back? Is this how the game is intended to work, or am I missing something?  Thanks!