No, I don't run a lot extra -- I have Origin, Steam, Razer Game Booster (I've tried it both with this enabled and not -- makes no difference) and a few system utilities. I sometimes am running Chrome as well, but that has no effect on the load times either.
I did not notice a signicant difference in CPU usage (about 45%) and memory usage (sits around 2GB) when loading and just playing. It goes up a little -- the CPU usage maybe goes up to 55% and the memory usage to a little over 2GB as opposed to a little under.
I tried deleting all the extra townies -- that made no difference either.
This save has probably close to 45 hours on it at this point. New games are fast.