A good way to attempt to rule out your ISP is to try and play over the internet on your phone.
Tether your phone (not connected to your WiFi) to your computer and see if you can replicate the issue.
Another thing to try to see if your ISP might be messing with your traffic could be to see if the issue happens if you a VPN (there's many out there where the first 500MB are free or something like that).
This will ofc. make your ping higher but if your Ex.Offset spikes go away this would indicate your ISP as the issue.