I am pretty late in my second generation matriarch (24 babies). I haven't repeated men and I also haven't had any trouble finding other men. My townie population seems to regenerate pretty frequently. However my matriarch doesn't keep relationships with men that don't actually live in the town so every so often I'll go into manage worlds and move 8 of them into a random house together.
I'm not too sure about the 180 townie limit. If there is one I haven't run into it yet but that may just be because my matriarch doesn't actually remember all of the people she has met.
