TL;DR
Your "solution" is not solving anything. It's making things worse. How about this:
Change expose to deal X%, capped at X (based on attacker power, matching pre-nerf haat levels)
Let the boss gain X% tenacity on each debuff/TMR attempt, not on each hit, and let it last to the end of the encounter instead of resetting
Let the Sith Lords ignore the irresistible keyword
Some reasoning:
Percentage mechanics can't be solved with coefficients. Half of infinity is still infinity. Next raid with twice the hitpoints will have the very same problem with expose again. What happens if level and gear increases make expose the dominant mechanic in the arena? Is it up for another nerf then?
The obvious simple fix for the expose problem would be to put a cap on the damage. You can put the cap on the HP on phase 2 in HAAT, and it wouldn't affect any old content, and it wouldn't be overpowered in the new one. Can we please get an explanation why this solution wasn't considered more carefully? The same should be applied to Deathmark, Thermals and all other percentage effects. You could even base the cap on the character power somehow to allow for character progression.
The stacking tenacity in itself isn't a bad idea, though the values are totally off and the implementation is wrong. The current solution makes it a one-off to use debuffs, and if one of your non-debuffers happens to go directly after the boss, he will block your debuffer's chance anyway.
The reasonable way of implementing diminishing returns would be to put it on the effect itself, not on something else. And not more or less randomly. If you want to stifle debuffs, increase the tenacity by less, do it on the debuffs (even resisted ones), and let it stick throughout the whole encounter instead of having it "randomly" reset. That way there could be strategic decisions on how to handle it, like stacking potency and/or maximizing usage of debuff teams before tenacity runs out of control.
As many has pointed out, the high tenacity makes all teams without irresistible effects working very bad or not at all. Rey teams aren't stronger than others due to expose damage, they're stronger due to being able to use debuffs at all. I assume the irresistible function exists in the first place as a counter to tenacity up, which in turn is a quite bad design. It would be a good thing to start looking at the root cause instead of trying to patch it with another bad solution on top.
The long term fix would be to sort out potency/tenacity once and for all. As a short term fix, you can simply allow the Sith Lords to resist irresistible effects. That would of course require a change in the boss tenacity, or the few teams that can do something now would be as useless as the rest.
As a side effect on fixing the irresistible TM reduction with a proper stacking tenacity solution, you can also remove the +speed on damage the bosses get. If you want them to get turns faster, it would be simplier to increase the +5% TM on taking damage than to add another mechanic to do the same thing circumventing the irresistible TM reduction.
