Had the same issue, I started with moving the game to my SSD drive because I had similar symptomps with another game (Dota 2) when it was on the HDD.
It didn't help much or at all.
Your solution fixed it completely though - thank you balling93!
I also saw a post from Killer employee in this topic (they really found this topic and replied to it - nice!) asking us to update the Killer drivers and report the result to them - will try that later.
For some people asking about the Killer Network Control - if you don't have it, you probably need to keep on searching for your solution..
Killer is a brand of network cards, and this is their driver/utility that allows control of the network card.
Check your Windows Device Manager if you have a Killer network adapter at all first.