On top of what Naraic said, it's the symbol on the mod that determines the bonus, not the stats. E.g., if you have 6 health mods with speed secondaries and a speed primary on the arrow, you get 3 times the health bonus for 3 pairs of 2. The mod stats apply on top of this, meaning you also get more speed even though you only have health mods.
With speed mods, you have to pick another mod type to get a second bonus. This can be either health, defense, potency or critical chance. Everything else needs a set of 4. In your case, it would make sense to farm another potency mod with speed secondary.