Forum Discussion

apostolateofDOO's avatar
5 years ago
Solved

Servers: Troubleshooting for EA/Respawn and Multiplay

Let's put the boot on the other foot...

I'll put myself out there and say i think it's pretty unanimous that performance isn't that great in the games current state that's only getting worse as each season is added and before i start, i would just like to say that this isn't a rant (your servers are trash) thread and networks, servers, code and game development aren't my area of expertise.

I'm only mentioning things I've read briefly (some of which will be copy/pasted from online sources for my own convenience) or mentioned through discussions here that could be a cause or possibly be a part of the problem and i understand from a backend point of view that this is a process of evaluating costs and benefits. We need some benefits now as more than enough users have given plenty over the past 2 years to cover those costs. No excuse when you charge 150+ for an heirloom that's 3x the price of a typical AAA title. If you're to be so bold to expect us to pay that amount shoving these events down our throats each season then you can bet i'll have the audacity to request a decent live service when it's pretty sub par in comparison to most FPS titles either current or in the past that don't charge anywhere near that amount.

I'm trying to find a better experience than what i currently have (beyond the realms of user troubleshooting that i have already done multiple times) and hopefully for others too. Let's exclude the bare min specs users playing on wifi or bugs like sound issues they can't replicate as that isn't what i'm getting at nor that games are ok for most users to play at 30fps when we live in an age of 240hz + monitors and EA, the global leader in digital interactive entertainment officially advertising Apex as a competitive shooter in tournaments with a large prize pool.

Now, some of the problems that we're experiencing could be because of some of these things mentioned or totally unrelated. Point of this discussion isn't for replies of why they can't do this or improve that, but a workaround, compromise or solution that might actually resolve or improve some issues for a lot of users that they (EA/Respawn or Multiplay) can do on their end.  So, if you apparently know what you're talking about then i don't want excuses.. Problems are there in abundance...

They need addressing. ASAP.

Matchmaking Server - Server selection - Ping limitations per match.

From my limited understanding, Respawn has a dedicated matchmaking server that puts you through to Multiplays Hybrid Cloud solution that can place you into different servers that's integrated with the 3 biggest public clouds, Google Cloud, Microsoft Azure and AWS in over 190 data centers but the cloud machines are more unstable, due to weaker processing power, differently tuned buffering, maybe less optimal/reliable routing, etc..

SBMM, region, ping and other metrics to determine where to put you in a match and they can also migrate players to other data centers because of these things or to decrease que time which they're trying to keep very low for obvious reasons also effecting your matches.  Developers want to provide a consistent, curated, experience for players while some players very much want complete control over their experience.

As selecting our data center and instant play (ready now) is our ONLY option which will take the above into consideration when joining a match, the priority of providing you a smooth experience will decrease with longer que times and the problem made worse in off peak hrs in your region. This doesn't just apply to you, but all 60 players in a match so once you wait x amount of time then it will fill these spots with players that have high ping from another country and possibly even another continent to you.

Playing with a high ping means game lags, which adversely affects the player's game. Most times, with high ping, many servers of FPS games automatically disconnect the player and kick them out of the game but we don't have this in Apex.  Latency is the time measurement of exchanging information (ping) from one computer to the server.

What makes this worse is every time you or i join a new game it could be with a different server or provider and also potentially under load effecting performance causing micro stutter, slow mo, hitreg and delayed response (getting shot through doors or behind cover) and whatever else. These will already exist but be made much worse and this inconsistency is infuriating for those playing the competitive mode with RP on the line when trying to climb in the higher tiers of ranked.

Here's brief explanation of high ping, peekers advantage and lag compensation.

Guide to playing with or against high ping in FPSs that have lag compensation (kovaak.com)

With our performance display with limits pre set by what Respawn thinks is acceptable like ping 170+ until it shows the ping icon for example, it would be nice to set our own limits with more information and have network round trip included like Valorant.

A quote from Dave Heironymus, technical director of Valorant at Riot Games.

“The Valorant team has put extensive effort into determining the best combination of tick rate and latency that will minimize peeker’s advantage, and those tests showed that a 128hz tick rate and 35ms (or less) latency would be best for our players.” and “Blind tests also showed that elite FPS players … can reliably detect when the game is running at a lower tick rate.”

Most Adept FPS players look for performance which has led to Valorants success as let's be honest the movement, abilities, characters and all round generally, Valorant is an inferior game to Apex with performance being its main attraction. If Apex goes along these lines it would bring some of those players back and be good for the game overall.

With the addition of series one ranked in Season 2 and it becoming more refined for competitions over time with tweaks to the RP system, legend balance and further map releases and changes, the foundations are there for Apex to rise in esports. Many more improvements can still be made to ranked that i'll push for but that's beside the point.. Having a server selector and instant play is doable and i think having the extra option or choice, especially for those climbing the higher tiers or potentially wanting to play Apex professionally is a good thing imo.

Perhaps that's complicated in the current structure through Multiplay (i don't know) but could that be changed? Having random and unstable experiences at that level is causing unnecessary frustration for those future stars that could potentially be advertising the game and end up sending them to other games that have more focus on aspiring competitive players.

Network, packets, servers and server tick rate.

We all heard this before and it's the worst amongst ALs biggest competitors. I don't understand the whole sending packets with pigeons but just remove the crap, streamline and optimise it. Same with the ''netcode''. Just as you can remove all the fliers, prowlers and such if it's draining server resources. Have that content in the game by all means for those that want PvE but if it's hindering competitive PvP matches then remove it or have it separate.

Most games that have upgraded server ticks has been noticed by most just as you would notice playing on 30hz TV then going to a 60/75hz or from that to 144/240hz. Also, give us the option for higher client CPU/bandwidth usage. Clients could just choose their own rates. You apparently have a 10 year plan? Invest in it.

Here's some old videos from Battle(non)sense that are still relevant today.
 
 
 

The most important part to take away from this is THEY CAN improve it but either the developers don't get the resources or don't feel the need to make these changes.

There is also the fact pay-to-enter servers like ESEA, FACEIT, and CEVO, the last of which has locations in New York, London, France, Sydney, and more, all run on 128-tick environments, but with the added catch of being paid services.

Rumour is (so take with a pinch of salt) Respawn have a 6 year contract with Multiplay for their hybrid cloud server solution. Now i understand the benefits of this originally to address scaling issues but now the player base has settled into what could potentially be roughly estimated levels for new seasons release and throughout, could this be optimised? Or have separate agreement/contract for Ranked servers?

Crossplay

Intentional limits to accommodate inferior hardware - I don't know how much weight this carries or any of the insides of this but have separate versions of the game if it's true or let us toggle it. This is assumed to be worse with switch this month and the already broken ''lag compensation'' in conjunction with AA (aim assist) sounds like it's not fostering a very fair competitive environment.

We already have enough exploits in this regard.

Edit: Recently particle FX count was lowered for Caustic Gas (tac and ult), Bangalore Smoke, and Gibraltar ult likely to accommodate Switch users and reduce GPU load.  

https://twitter.com/GH057ayame/status/1367702094426173442?s=20

Word was around that other features had been changed long before this that weren't included in the patch notes and likely more in the future too.  It was mentioned in a separate thread that Respawn can tailor the game to specific platforms so why they have done this across the board that will negatively effect most users is unknown.

One size clearly doesn't fit all.

Code refactoring

It's always done to optimize things, it's usually happening where/when the code gets too messy because of a lot of patching or when there's new tech or device related features available that the game could benefit from. As the server issues are more prevalent when a new season or large update comes out which usually don't get fully fixed, i would imagine the pressure of keeping up with these content update schedules has had a negative impact on the game over time and while i appreciate they will constantly work on this, it's evident that many QoL issues have been neglected because of time, effort or resources required to fix them properly.

Lower the amount of players in a normal match, different game mode or LTM.

I think most have noticed a sluggish response at the beginning of the match on all platforms at some point. This issue has been prevalent since release when it was a very common occurrence and although it has improved since then (even if it's minor, credit where it due) we still have the same problem occasionally. This is also tied to the CODE: X,Y and Z names Respawn gave to address where the problem is so they can locate and fix it. Due to lack of communication we don't know what progress has been made but it still occurs, even if infrequently. 

Just to note, Respawn have highlighted in Trello about the slow matches that's been happening a lot recently. 

I'll keep updating this but feel free to add anything in the meantime...

An article that mentions Respawn are aware and that they'll address or at least communicate about servers issues.

Apex Legends Season 8: Fans Upset With Servers and Connection Issues (dbltap.com)

147 Replies

  • apostolateofDOO's avatar
    apostolateofDOO
    5 years ago

    ''In recent seasons, we’ve begun leveraging the help of our awesome data science team to batch and crunch (i.e. gather and analyze) one week of data at a time to detect excessive packet loss and server performance issues. This approach has been paying off already.''

    Nope. I played last week and you still have dog poopoo servers.

    ''Let's go through various scenarios with two players in Apex Legends called HIGH and LOW. Let's give HIGH a high ping of 300 ms, and LOW a low ping of 50 ms. The difference in their pings is 250 ms.''

    Good players response times are lower than this and you should not have a 300ping player in any of your matches, at all, in competitive modes. Casual is fine...And the explanation still doesn't explain why we get shot full seconds at times behind cover, not milliseconds.

    ''At the end of the day, the server is a sort of time machine. It constantly rolls back the world state to see if your shot hit someone, and then updates the world for everyone accordingly.''

    Reduce the amount you have to time travel with a ping limit. Simple really.

    ''Let’s talk about hit registration. A “no reg” or no-registration of a shot means you think you hit your target but the server basically disagreed. From your perspective, you get all sorts of confirmation in the form of blood spray and sounds, but no damage counter shows up. In a shooter like Apex Legends, this is extremely unpleasant. 

    It can happen for a multitude of reasons. Sometimes, high latency or packet loss can cause your local simulation to become slightly out of sync with the server. You shot where you saw someone, but actually you were shooting where they had been previously. Unfortunately, you don’t find that out until your version of the world catches back up.

    Sometimes, it’s just a bug with the game’s physics simulation.''

    A lot of players have good connections to the closest data center and considering it gets worse with each seasonal update that effects everyone then my money is on your simulation being the issue.

    ''Okay, so we’ve discussed some possible downsides that come with increasing server tickrate. But what about the upside of going from, say, 20Hz to 60Hz? Come on, Respawn! Wouldn’t that make the servers three times faster and three times better? Just do it!

    Based on our findings, it would not result in a meaningfully different experience, and we want to explain why. 

    For the sake of the argument, let’s assume that you’re averaging about 50ms ping, or latency. Remember that your ping measures the speed of a full round trip between your machine and the server. So assuming there are no other problems like fluctuating latency or hardware lag (eg. display devices introduce 20-50ms delay), the server will receive your input 25ms (half ping) after you press a button or flick your mouse. 

    Put all this math together, and you realize that 20Hz servers result in about five frames of delay, and 60Hz servers result in three frames of delay. So for triple the bandwidth and CPU costs, you can save two frames worth of latency in the best-case scenario.''

    You assume everyone plays on last gen consoles and TVs from the 90's. Some have 144-240-360hz monitors with 1ms response time. Redo the math for top of the line hardware console or PC players and come back with a reasonable excuse instead of catering to just casual last gen console players living in the amazon, arctic or deserts playing on Starlink wifi when you have a growing competitive e sports scene.

    This is why, throughout this blog we’ve shared a number of improvements that we’re pursuing in the near future, including:

    • Using real-time alerting that will allow us to identify problems and respond more quickly
    • Implementing tools for identifying servers so we can remove and replace problematic servers rapidly
    • Focusing on slow-mo servers—removing problematic servers is one step, but our goal is to make this drastically less common with code changes
    • Reducing latency with better optimization of new features
    • Fixing hit-reg bugs and building automated detection tools to help us avoid introducing new ones

    Reducing latency and better optimisations is something you would have been working on since release. You mention no real improvements coming except the removal of the odd slow server and not how your game runs poorly in general.

    You also forgot to mention how SBMM would increase yours, or other players in your matches ping, putting me in eastern european servers (have london selected) who also have players from South Africa. Wonder why i get shot through doors...

    CLOSING THOUGHTS

    You make excuses for a one size fits all matchmaking for inclusivity that you conveniently left out and compare your improvements to consoles as the investment will cost you money which a lot of PC and next gen console players would see tangible benefits from but because the majority of your players wouldn't benefit or notice, it's not worth the extra cost to you.

    @DoYaSeeMe ''Other modes - this divides the playerbase. They know this first hand, from Titanfall 2, that's why they avidly avoid it.''

    We have a new 3v3 mode now. They could easily increase it for that or would 6 players be to much for the servers because of bullet physics? Yes it would take more processing on their end but it's technically possible when snapshots are of a small map with 6 players...

  • Littumulkero's avatar
    Littumulkero
    5 years ago

     @apostolateofDOOM 

    Thanks for this post. I hope this gets more views since season 9 launch was a bit harsh. I agree that there must be something that can be done to improve overall situation how networking is working in APEX.

    @DoYaSeeMe 

    Thanks for your replies and comments. Earlier on you did write possible solutions and race car analogy. To me these ones were great since they pulled me towards understanding what situation might be on the other end. Dunning-Kruger =).


    I dare to claim that improving game general stability and errors would really help in the long run. They have made it. Yes. There still are some issues that are from season 1. I personally would be happy to play with same legends and everything for at least half a year to make most of the occuring bugs to go away (missing sound, code.net, no reg, packet loss).

    In the end keep up gaming and thumbs up for Apex Legends.

     See you in arenas..when servers are up. =)

  • @Littumulkero Thank you.

    I'll be updating this thread soon with some feedback to their server post and highlighting where some actual problems are and what most definitely can be improved upon. I'll even do some number crunching for them. 😉

  • apostolateofDOO's avatar
    apostolateofDOO
    5 years ago

    Quick bump to remind myself this thread still exists as do all the other problems mentioned that haven't been addressed.

    Will get round to it when i get some free time.

  • Second server search result... Wow, I doubt they will do anything if posts like this get put on the back burner... It must be all the players internet connections dropping coincidently at each update. Once is happenstance, twice is coincidence, but 3 times (or in this case, 10 seasons and many broken updates inbetween) it's just neglect or incompetence which they try and deny and always pass it to the player with Troubleshooting. They seriously need to take responsibility and communicate what they're going to do to improve this.

Featured Places

Node avatar for Apex Legends General Discussion

Apex Legends General Discussion

Discuss the latest news and game information around Apex Legends in the community forums.Latest Activity: 26 minutes ago
31,728 Posts