@OldTreeCreeper Servers work differently than the usual client PCs. Normally, they host more matches, from dozens to hundreds, depending on the game. For this, they use a lot more CPU cores and a big amount of RAM. They don't do the rendering part, so most things related to graphics aren't loaded. It's just the hitboxes and coordinates that they care about primarily, to calculate interaction, positioning, maybe some physics, etc.., other stuff like textures and sounds are just id's that they pass back and forth (so that player X know what skin is used by Y, what he pings or what quip he's playing, for example).
It's all about processing, so the typical defragmentation would have hardly any effect on the server. Defragmentation is used to improve the speed of reading data from the drive, usually to load it to RAM, but game servers load almost everything they need at the start of the game, otherwise they would be very unstable under load.
Because the number of players in an Apex match is big, this can make processing time take a lot, but also require more data to be sent per update. These go down significantly as players get eliminated, which is why the second half of most matches feels much smoother.