The thing that gets me is when I know my putting stroke was slightly off center on a short putt with no meaningful break but the ball goes way off to the side. Doesn't happen often, but it's alarming when it does.
Pros never miss by much to either side of the hole on short putts with no meaningful break, so I don't think there should be that much of a direction penalty on short putts. Unless you do something really crazy during the stroke, the degree to which a short putt misses should mainly be based on how much your aim was off (taking the break into consideration) and how much the weight of your putt was off.