So confused, what am I doing wrong??

I collected all of the valentines in my town, an hour later I checked back and one of my brown houses had a handshake. I can only surmise that there were no cards available, so there may be some additional lag time for regeneration beyond the town owner collecting.