I think it partly has something to do with the time of day, and the fact that most everyone has more nerds to attack with. I've had that happen to me a few times here and there, but I just move onto the next town when it moves directly to a burning castle.
Replies