Early on, I'd focus on health mods and other sets that only require 2 mods for a bonus (like critical chance or potency). Unless the character's base speed is very low, you can already get above 200 speed using non-speed sets by picking arrow mods with a speed primary and other mods with speed secondaries. You'll need the extra health at lower gear levels, as well as any protection bonuses from mods.
If the recommendation calls for critical chance and damage, throw 2 crit. chance mods in with your health sets, or 4 if your character's base health is high enough. That way you can still up your health a little. E.g., Vader is very beefy for an attacker, but other attackers are still squishy even at gear level 8 or 9. Why waste 4 mod slots on more crit. damage or speed mods (which may actually be slower if they lack good speed secondaries) instead of upping their health by 20%.
It's also easier to systematically level up mods to 3 or 6 and check for speed bonuses when you prioritize one type, and ideally one that only needs 2 for a set. You can still switch to better mods later on and hand down your old mod sets to characters you only use in territory wars/battles.