I usually play with none up at all unless I am building. If so, I put them all the way up to do the wallpapers and put up wall decorations then get taken back down,
I usually only play with partially up walls unless, like others have said, something is blocking my view or I want to take a screenshot with a specific wall in the background.