Forum Discussion

Re: WHAT MAKES APEX TICK: A DEVELOPER DEEP DIVE INTO SERVERS AND NETCODE

That was indeed an interesting read.

I have to ask though if the service providers provite inadequate service, at what point do you say that's enough we don't want your second class servers any more? Does apex use aws like fortnite and other games? If not why not?

How exactly are players expected to feedback on all the points mentioned in the blog, as suggested, is there a central hub for this or will an ingame feedback be developed? 

3 Replies

  • DoYaSeeMe's avatar
    DoYaSeeMe
    5 years ago
    @OldTreeCreeper There are some important things missing from the article. For example: of all the issues that hurt the experience, how much is servers and how much is network / client device? Is the failure rate of the data center hardware normal or higher than it is for other providers? Players always ignore the myriad of issues caused by the network and their own devices and sowftware.

    Respawn uses AWS since Titanfall, but they are also using Google Cloud and possibly Microsoft Azure as well. Multiplay has access to a lot of data centers (over 170) and has hosted a lot of games so far, I doubt they are providing second class servers.

    I think the actual problem lies with the high computational power needed by Apex in particular.

  • OldTreeCreeper's avatar
    OldTreeCreeper
    Hero+
    5 years ago

    @DoYaSeeMe I can only go by what the blog says for now. Taking 2 weeks to find a faulty server and only if reported and Investigated, would leave a lot to be desired in any business model.

    I'm not ignoring the other issues, just discussing a particular one, that they have voiced an interest in improving.

    I have in the past voiced concern here in this forum about lag, and been told on more than one occasion that the problem is on my end, despite my WiFi being fine. Perhaps we will now see a change in that absolute problem solving, now that the above blog accepts that there is in part a problem on the server end.

    You are correct that we don't know the fact and figures of it all, we will have to wait for that info, I guess. 

    What I can say is that I have played a lot of apex and fortnite over the last 2 years. There is a significant difference in the servers when I compare the 2 at a given time and run tests with ookla on my WiFi to be sure of my end.

    Perhaps it's the mix of servers that is part of the problem, rather than just the one host.

    I can only imagine the computational power used by apex as I have had a small insight with yt learning to use unreal engine. But I struggle to understand why someone with a wired connection with a 25 ping would have similar problems to me, unless the computational power needed is too much at the mixed server end, in which case a compression of sorts is desperately needed. 

  • apostolateofDOO's avatar
    apostolateofDOO
    5 years ago

    As we got a response i pushed for, i'll let the other thread die out and create another in the near future for changes that can be made. Yes, i'll even sit and do the maths myself if i have to. *sigh*

    In reply to this that i also put in my other one,

    ''In recent seasons, we’ve begun leveraging the help of our awesome data science team to batch and crunch (i.e. gather and analyze) one week of data at a time to detect excessive packet loss and server performance issues. This approach has been paying off already.''

    Nope. I played last week and you still have dog poopoo servers.

    ''Let's go through various scenarios with two players in Apex Legends called HIGH and LOW. Let's give HIGH a high ping of 300 ms, and LOW a low ping of 50 ms. The difference in their pings is 250 ms.''

    Good players response times are lower than this and you should not have a 300ping player in any of your matches, at all, in competitive modes. Casual is fine...And the explanation still doesn't explain why we get shot full seconds at times behind cover, not milliseconds.

    ''At the end of the day, the server is a sort of time machine. It constantly rolls back the world state to see if your shot hit someone, and then updates the world for everyone accordingly.''

    Reduce the amount you have to time travel with a ping limit. Simple really.

    ''Let’s talk about hit registration. A “no reg” or no-registration of a shot means you think you hit your target but the server basically disagreed. From your perspective, you get all sorts of confirmation in the form of blood spray and sounds, but no damage counter shows up. In a shooter like Apex Legends, this is extremely unpleasant. 

    It can happen for a multitude of reasons. Sometimes, high latency or packet loss can cause your local simulation to become slightly out of sync with the server. You shot where you saw someone, but actually you were shooting where they had been previously. Unfortunately, you don’t find that out until your version of the world catches back up.

    Sometimes, it’s just a bug with the game’s physics simulation.''

    A lot of players have good connections to the closest data center and considering it gets worse with each seasonal update that effects everyone then my money is on your simulation being the issue.

    ''Okay, so we’ve discussed some possible downsides that come with increasing server tickrate. But what about the upside of going from, say, 20Hz to 60Hz? Come on, Respawn! Wouldn’t that make the servers three times faster and three times better? Just do it!

    Based on our findings, it would not result in a meaningfully different experience, and we want to explain why. 

    For the sake of the argument, let’s assume that you’re averaging about 50ms ping, or latency. Remember that your ping measures the speed of a full round trip between your machine and the server. So assuming there are no other problems like fluctuating latency or hardware lag (eg. display devices introduce 20-50ms delay), the server will receive your input 25ms (half ping) after you press a button or flick your mouse. 

    Put all this math together, and you realize that 20Hz servers result in about five frames of delay, and 60Hz servers result in three frames of delay. So for triple the bandwidth and CPU costs, you can save two frames worth of latency in the best-case scenario.''

    You assume everyone plays on last gen consoles and TVs from the 90's. Some have 144-240-360hz monitors with 1ms response time. Redo the math for top of the line hardware console or PC players and come back with a reasonable excuse instead of catering to just casual last gen console players living in the amazon, arctic or deserts playing on Starlink wifi when you have a growing competitive e sports scene.

    This is why, throughout this blog we’ve shared a number of improvements that we’re pursuing in the near future, including:

    • Using real-time alerting that will allow us to identify problems and respond more quickly
    • Implementing tools for identifying servers so we can remove and replace problematic servers rapidly
    • Focusing on slow-mo servers—removing problematic servers is one step, but our goal is to make this drastically less common with code changes
    • Reducing latency with better optimization of new features
    • Fixing hit-reg bugs and building automated detection tools to help us avoid introducing new ones

    Reducing latency and better optimisations is something you would have been working on since release. You mention no real improvements coming except the removal of the odd slow server and not how your game runs poorly in general.

    You also forgot to mention how SBMM would increase yours, or other players in your matches ping, putting me in eastern european servers (have london selected) who also have players from South Africa. Wonder why i get shot through doors...

    CLOSING THOUGHTS

    You make excuses for a one size fits all matchmaking for inclusivity that you conveniently left out and compare your improvements to consoles as the investment will cost you money which a lot of PC and next gen console players would see tangible benefits from but because the majority of your players wouldn't benefit or notice, it's not worth the extra cost to you.

    @DoYaSeeMe ''Other modes - this divides the playerbase. They know this first hand, from Titanfall 2, that's why they avidly avoid it.''

    We have a new 3v3 mode now. They could easily increase it for that or would 6 players be to much for the servers because of bullet physics? Yes it would take more processing on their end but it's technically possible when snapshots are of a small map with 6 players...Although i still believe both modes would benefit from 30 at absolute bare minimum.