I don't see what I do in the voting process. Mostly I play vanilla but have a few mods I'll use now and then to play a certain way (think Road to Fame types). I pull them out when I play other saves especially if I have a number that makes one save work. I've started naming my save after the mod(s) to avoid confusion for me. I like to test mods this way too.
I don't really like to have to keep up with the updates so.. I don't feel it ruins my whole game if I don't have this or that mod for awhile. I just play my mod-less saves that way and sometimes end up enjoying the game differently by doing this. Then I miss certain mods and the enjoyment cycles around again when I add it back in or try another. I do really like how easy it is to add in or remove mods now.