unable to harvest nightly mushrooms?

I have tried several nights in a row to harvest the nightly mushrooms up in the ruins area. I've tried spending the day nearby and also just arriving at night. Not having any luck. Any ideas?