I guess it depends how many you have? I'd say at any given time, I have 2-4 oversized plots going. Usually I'm shooting for one oversized for the festival, two normal, and one small. (The normal and small are for some combination of eating, selling, and canning.)
I have found that typically one dose of super vitality fertilizer and one round of encouraging growth is usually sufficient to get an oversized crop. (Although I do also have bees and rabbits most of the time, so that could be a factor as well.) One dose of regular vitality fertilizer will generally make normal sized crops with the oversized seeds. For the ones I want to be small, I don't apply fertilizer or encourage growth, but I do weed and spray for bugs (unless my bird and rabbit helpers get there first).
Long story short, it's not so bad, but having helpers makes things better, and it will obviously depend on how many you have going at one time.
(For what it's worth, my typical small farm/home garden household also has most of the berry bushes, one fruit tree, and 4-6 regular/staple crops. Sometimes also herbs.)
