So confused, what am I doing wrong??

If you visit a neighbour that has no cards in his/her town, it means that other neighbours visited already and tapped all the cards. As long as that neighbour does not collect these cards, everyone else has to wait.
If I come into a town with no cards, I'll check back later. You have to remember, not everyone else is living in the same time zone, so it is possible that you will have to wait until they wake up and start playing.