In my opinion there's not a one size fits all answer. Very few characters benefit from losing speed, but there are a couple of corner cases where you might prefer it. It really just comes down to how valuable the primary is to the particular character.
E.g. I have seen plenty GG with a health primary arrow over speed since its of benefit to his DPS. In that case, health primary with a good speed secondary is probably not an awful tradeoff.
In 99% of cases the gain at the expense of speed loss is probably not worth it though. Keep the mod though, 20 speed secondary is still nice.