Xbox Packet loss only on AU (sydney) servers

Had this issue last month or 2 where I can play on Aus servers, ping can be fine around 40-50 but constantly having packet loss. I switch to Tokyo servers & ping is 200+ but no more packet loss. It's only on my internet as I used Hotspot data & packetloss went away on Aus servers. Any fix as my ISP said everything is fine from there end