Also worth mentioning, ISP's bandwidth management and traffic prioritization. For example: who's paying better, streaming/download services or game publishers? Many people watch Netflix, Youtube or TikTok at the same time, which takes a lot of bandwidth. So who's data comes first? Most servers aren't NASA's super computers. But this can also be applied on the users end: https://www.howtogeek.com/75660/the-beginners-guide-to-qos-on-your-router/
There are also server checks. Even a ping might get verified (firewall and other security checks) before returning, to make sure it's not some malicious intent like ddos or a hack attempt. There are a lot of pings going on internet-wide.
Lastly, many forget that internet traffic is like regular traffic. The more people online doing bandwith-intensive activities, the more congestion. A server can only handle this much. We're not in Star Trek era, yet. So I guess all we can do is to accept and live with or mitigate these issues in some way and await improvements as the time goes on.
Edit: forgot to mention the speed limit of electricity and conductor material resistance. It's basic physics.