Mod TFP for crit chance. He needs to crit to apply tenacity down.
Fracture the boss as often as possible. On raid bosses fracture decreases their speed (not stacking with Vader's speed down) and reduces their counter chance (irrelevant for Rancor).
While the boss is fractured fire away with specials (especialy if Thrawn's Ebb and Flow is zetaed). However, save Tarkin's AoE for when the boss is not fractured or for emergency situations (at least until you are comortable with the raid and sure that you can solo it all).
Save Vader's Saber Throw for when his Force Crush will be available next turn. This is to quickly reapply the speed down, which is also removed by Saber Throw. If you use Saber Throw while the Rancor is toppled, the damage will be increased due to the various buffs you get. You could also time saber throw with the offense up from EP.
This is all just until your team gets good mods and gear after wich you can auto the whole thing.