It happens to me only on certain lots, for example The Summer Home lot in Windenburg and on Straud Mansion in Forgotten Hollow. Every time my Sim travels out of the lot and comes back the plants get reverted to either dirt piles or sprouts.
Some lots, for example Potters Splay in Willow Creek work perfectly with garden, at least for me. My 100 baby challenge family lived in Potters Splay lot for a long time and their plants were fine the whole time, but when they moved to The Summer Home lot and the matriarch traveled to another lot and came back all the plants had reverted to dirt piles. They grew back, but as soon as she traveled again, the same thing happened.