There are ways to make this easier, and there are also a limited number of toons for which swapping is worth the trouble. I don't swap mods for TW or TB - like Zanir - though I do for raids. I like my raids to be as smooth as possible, so I'm willing to take my best mods off my GLs and drop them on raid toons and then put them all back later.
It's up to you to figure out what situations are worth changing mods for. But if you need help in figuring out load outs, which allow you to swap in or out an entire set of 6 mods at once, someone can help you with that.