Skipping Replays

I skip most of my replays during a match. I have played against some opponents who don't, and I have to watch their replay. So when that happens I don't skip my replay, because I figure if I am having to sit through my opponents replay then my opponent can sit through mine. For some reason, my opponent manages to skip my replay. Then if my opponent scores again I'll press the button to skip the replay and it doesn't skip and I  still have to sit through my opponents replay.