Roughing the kicker/punter

I don’t mind penalties but it never fails when I’m playing a superior team and finally stop them on fourth down and they punt,  somehow magically my defense breaks through and roughs the punter. It is so amazing. Never happens when I’m playing a lower graded team nor do they do it to my punter!!  It’s like the better team brought their own refs with them into the game. Wait there’s more!  lol. After they get their first down they either score or get a chunk play. I’ve learned to call dime or prevent for the first play after the first down. They still get a lot of yards but don’t always score on me.