If you click on info, it breaks down the banners for you.
You get a huge bonus for beating a territory. So your opponent got an extra 300 banners just for killing your top territory where you could not. Subtract that off and they are 56 banners behind you. Given the difference between setting defense and max victory, they may have fought more efficiently than you. You get bonus points for unused slots, full health, full protection, etc.
Their score (assuming your bracket has the same scoring system as mine): Def: 100+90+90 + Terr.Bonus: 300 + 155 = 735. (1120-735)/7 = 55 average banners per battle.
Your score: Def: 100 + 90*6 + Terr.Bonus: 155 = 795. (876-795)/2 = 40.5 banners per kill. By any chance, did you take two teams or fleets to defeat one of theirs? You lose a bunch of banners for the first try bonus.
But yeah, if you had beat the other team and conquered the other territory, you would have still won just due to your opponent losing banners from setting defense.