You've had 3 draws? Wow!! I've been playing since the start of GA/GAC and I don't think I've ever had one. I'm not sure which is more surprising, you have had 3 or me never having any. I did lose 3 in a row recently all by less than 6 banners, though. Super grrrrrrrrrrr!
I think you're right. Getting even banners with a smaller roster is more of an achievement, and the smaller roster should be given the tie break. However, the end of the day, don't fret about it. You'll win as many games as you deserve.
Sometimes, losing the last game of a season will give you more wins in the first week of the new season. You'll get 3 easier games than you would have if you won that one.