Start climbing early. If he's ahead of you, jump to the shout right in front of him so he can only knock you back one spot.
Then do your next attack at random intervals. Do you think this guy is going to sit in arena and constantly check for your movement every few seconds for a consecutive 20-30 minutes? If you're in the twenties or below his refreshes won't see all the available spots and he'll have to use more time to check for your movement.
If you wait until the last minute he can predict the timing of your moves. If you start moving earlier your have a huge window.
