I really enjoy building without the CaSt. In the Sims 3, I would spend so much time recoloring everything, I couldn't help myself, I had the impression my build where not totally finished if I didn't do it (and as there was not so many preset options, you needed to do it). It was a nightmare ! And as your game as to load a specific generated texture for each object recolored (instead of preset texture files), it makes it slower (to load, at last).
Yep, sometime, I would like to have more color options in the Sims 4. But most of the time, I'm happy wth the colors and patterns, and I think it's even more realistic this way : a specific piece of furniture, in real life, only exist in a few variation. And except if you take two items from a set, nothing perfectly match. It's so much easier and faster to furnish a build in the Sims 4.
It can be a bit more problematic in CAS, but if I really need a new color for a shirt or anything, it's easy to create a new swatch for an existing item with the Sims 4 Studio, so I don't mind.