Re: Servers: Troubleshooting for EA/Respawn and Multiplay

@apostolateofDOOM playing ranked with yt just now in our first game. He is on his phone hotspot and I'm on the usual mobile broadband. Enter the match and get as far as the jumpship, then got thrown out tried to reconnect but no. Last game summary shows as the previous match I played, not the one I was ejected from. Then message comes up that member of my party  ie yt disconnected, so I get a 10 minute penalty! A first for me. Why would I get kicked out of the match if yt got dc? 

Yt comes running down corridor saying that he got dc and then 2 code errors, code leaf and code net. And he also has a 10 minute penalty. 

@OldTreeCreeper I understand you both getting a penalty if you were premade but as to why you were thrown out if on separate connections, i don't know.

I played a little ranked yesterday and got from plat 2 to diamond and bullet mode from either DDOS or dog poo backend issues really shows this game up. It's not just the dead or ddos'd servers but the inconsistency between matches which is sometimes fine and playable but it's clearly not putting me in or even around UK half the time, for matchmaking i assume. Weirdly, connection shows it's always around 15 ping and everything else really low with no packet loss but runs complete garbage and only changes (for better or worse) when i join a new match.

@apostolateofDOOM separate connections from seperate providers. I'm on xbox, yt on pc. The only conclusion is that this error happened on the server end. That's why I mentioned it here.

I'm usually on around 70-80 ping with 0-2%pl, I'd be a predator at 15 ping 😃

@OldTreeCreeper Nah, you can't abuse the ''intended latency features'' then. 😉

How stable does FN play? I never played it but it sounds like they don't have nearly as many issues as Apex when it comes to backend servers and such.

It's still shocking how different the game feels each match.  I played a little bit last night and half way through D3 now, solo hot dropping and w keying into fights so you could probably make masters if you play for points and push for it. I'm playing really badly compared to before (more risk taking and slow reacting, just playing quite casually with a beer 🍺 😃) so if i can slowly climb when hardly playing then anyone dedicated with the time or premade should be able to now. 

They really should give pred a higher RP cost compared to masters now we have more obtainable RP and remove this top 750 per platform rubbish. It should split diamond and master between those that can climb and those that barely got there making it better for everyone in their respective tiers and encourage them to play more if their goals are potentially obtainable.

@Vdstrk You're welcome. I'll be adding more in the near future but making the most of Easter break and half term while kids are off and the suns out.

Rare sight in the UK. 😛

@apostolateofDOOM what "intended latency features"? I'm usually dead before I come round the corner and before I know it 😃

Got to platnimum last night and that's it. I just can't compete with my Internet ping, nor my age related decomposing slowing decision making process, too much like hard work, esp on the olympus map. For some reason I find ranked much easier on kc or we. 

I have played on holiday on a wired 25 ping and there's a world of difference. 

YT wants to get to plat for the first time and only has 500 points to go. I'm helping him avoid the suicidal numpties that are trying to remove his hard won points. 

Very rare to get a problem with FN, and never kicked out of a match. I would get a similar lag but it dosnt seem to make that much difference. I don't build and take advantage of the expectation of that with grenades. Play duo's with yt and ytlett with much success. It's just good of fun.

@OldTreeCreeperPriority given to the shooter but also high ping players /lag compensation ect. It's covered a little bit in the vids by battle(non)sense and why i have an issue with the one size fits all matchmaking sending me all round Europe and probably even further. I've only heard about people using it and other stuff with their connection to gain an advantage (streamers too with diego and others) but never used anything like that myself or really bothered to learn how they do it as i want a smooth experience even if it means i get it handed to me.

People say there's thousands of miles of cable but i shouldn't be getting sent through thousands of miles or 10 different countries across oceans and borders for a match when more than enough people play in the UK or at worst sending me to Germany, especially with crossplay and we're talking a few hundred miles at most, excluding auth servers ect and w/e else that shouldn't be effecting my direct gaming line.

I would offer to help YT but i went all the way back down to bottom of D4 as tipsy whiskey hot dropping wasn't favouring me tonight and would probably risk his points with my twitchy w key finger. xD

I added an article at the end of my post that was created around when this was made just saying Respawn are aware of said matchmaking/server issues and will be addressed or communicated this season that was also promised in the AMA on reddit so it shouldn't be long now before we hear something. I'll get round to adding some more to this hopefully in next couple days now kids back at school.

https://www.dbltap.com/posts/apex-legends-season-8-fans-upset-with-servers-and-connection-issues-01eyew9z10kt

The link to the reddit post i was looking for previously too.

Respawn here. We just shipped Season 8 - Mayhem. Ask us anything! : apexlegends (reddit.com)

@apostolateofDOOM soon as I hit plat, there were plats everywhere much to yt's discontent, as generally in gold we just had to deal with gold's. If we added diamonds to the mix, he'd blow a fuse. 😃

He would love to play with you sometime, perhaps in pubs when you are feeling favoured 🥳 and you can show him how it's done and I can watch you play after I'm dead 🙌

@OldTreeCreeper I'll be playing a bit at beginning of S9 for this new mode so i should be warmed up then but if he needs to climb before i can always play the better halfs acc as she doesn't use it, if i can remember the login. It'll be bronze but won't take me long to get to his rank.

@GRiPSViGiL I carry the torch with pride and in it shall hold the eternal fire of DOOM!

So we have an update with only 2 weeks left to go! 😃

Ryan K. Rigney on Twitter: "Have been working with Apex engineers on a “servers, net code, and tickrate” dev blog. It’s coming in two weeks and: * Acknowledges and explains the biggest issues * Explores recent/upcoming improvements * Contains a likely controversial but interesting rant about tick rate" / Twitter

@apostolateofDOOM here's a new take on the servers issue. Playing ranked again with yt and he is on his hotspot and the data ran out at 12, he was dc and penalised 😃 I'm sure he learned a lesson there!

So on to the main story. He joined me on our main WiFi after his 10min was up to play ranked again and what happens in the first match, I get chucked after about 2 min, but yt dosnt! I try to rejoin but it comes up with an error saying check my Internet connection!!! Clearly as yt could continue playing there was no Internet issue this end. Riddle me that? Not sure if I got a penalty as it took a few minutes to reconnect and I was distracted. 

Only thing I could surmise is that this type of server error has a built in default description for the end user of "it's your fault!" 

@apostolateofDOOM I have to wait for the neighbourhood to go to bed 😃

@OldTreeCreeper You using their wifi? 😃

Just a reminder (for myself mainly 🥳) we don't have long left for this announcement. 

@apostolateofDOOM the whole neighbourhood uses the same mobile pole, so there's a bit of congestion at times.

And no not my neighbours WiFi, the neighbourhood!! Do you need glasses 😃

What announcement, did I miss somthing? 

@OldTreeCreeper  Maybe you were siphoning the connection from a few different households with unsecured connections but had to wait till they were in bed so not to be seen with tree logging in... 😃 I really should stop posting online when drinking and not paying attention but... 🥳

I meant the dev blog in a few days.

@apostolateofDOOM none of the neighbours connection are close enough to use, even my own connection needs an extender to reach the sitting room from the kitchen!

Notwithstanding the time of day signal issue, indeed I would have to go logging to get a clear line of sight to the pole. When it rains the signal decreases. When the leaves come back on the trees, the signal decreases. When theres leaves and rain I'm pulling out my hair 🥳

@apostolateofDOOM started to play a plat ranked game with yt and I got immediately kicked out to code leaf. Game summery shows previous pub game. Don't think I got a penalty but yt is very  perturbed but did well 😃 he came 5th. His other fellow squaddie just left him at the start of the match when he saw only 2 in the squad. 

@OldTreeCreeper I read that to far ahead in my mind and thought you were going to finish with pulling your leaves out. 😃 Gz to him. Is he still playing ranked? I haven't touched Apex since i last mentioned and gonna be rusty for S9, again.

Have they identified or given a reason for all the code:net/leaf/shoe ect.. yet?

@apostolateofDOOM no idea on the why?

I haven no leaves or hair left to pull out 😃 yt has 192 point to go to plat, so hopefully we'll get there shortly and I can go back to kc 😃

A level 38 octane that had the 20 bomb, plat 1 with 178 kills just added to his score, staggering accuracy.... 

@OldTreeCreeper I don't and have never used use wifi so i won't guess but have you tried a wired connection? 😛

Well we got our post, finally: https://www.ea.com/en-gb/games/apex-legends/news/servers-netcode-developer-deep-dive

Anyone have any thoughts or something they want to add before i begin? 👿

@DoYaSeeMe I see you! 🤓

''In recent seasons, we’ve begun leveraging the help of our awesome data science team to batch and crunch (i.e. gather and analyze) one week of data at a time to detect excessive packet loss and server performance issues. This approach has been paying off already.''

Nope. I played last week and you still have dog poopoo servers.

''Let's go through various scenarios with two players in Apex Legends called HIGH and LOW. Let's give HIGH a high ping of 300 ms, and LOW a low ping of 50 ms. The difference in their pings is 250 ms.''

Good players response times are lower than this and you should not have a 300ping player in any of your matches, at all, in competitive modes. Casual is fine...And the explanation still doesn't explain why we get shot full seconds at times behind cover, not milliseconds.

''At the end of the day, the server is a sort of time machine. It constantly rolls back the world state to see if your shot hit someone, and then updates the world for everyone accordingly.''

Reduce the amount you have to time travel with a ping limit. Simple really.

''Let’s talk about hit registration. A “no reg” or no-registration of a shot means you think you hit your target but the server basically disagreed. From your perspective, you get all sorts of confirmation in the form of blood spray and sounds, but no damage counter shows up. In a shooter like Apex Legends, this is extremely unpleasant. 

It can happen for a multitude of reasons. Sometimes, high latency or packet loss can cause your local simulation to become slightly out of sync with the server. You shot where you saw someone, but actually you were shooting where they had been previously. Unfortunately, you don’t find that out until your version of the world catches back up.

Sometimes, it’s just a bug with the game’s physics simulation.''

A lot of players have good connections to the closest data center and considering it gets worse with each seasonal update that effects everyone then my money is on your simulation being the issue.

''Okay, so we’ve discussed some possible downsides that come with increasing server tickrate. But what about the upside of going from, say, 20Hz to 60Hz? Come on, Respawn! Wouldn’t that make the servers three times faster and three times better? Just do it!

Based on our findings, it would not result in a meaningfully different experience, and we want to explain why. 

For the sake of the argument, let’s assume that you’re averaging about 50ms ping, or latency. Remember that your ping measures the speed of a full round trip between your machine and the server. So assuming there are no other problems like fluctuating latency or hardware lag (eg. display devices introduce 20-50ms delay), the server will receive your input 25ms (half ping) after you press a button or flick your mouse. 

Put all this math together, and you realize that 20Hz servers result in about five frames of delay, and 60Hz servers result in three frames of delay. So for triple the bandwidth and CPU costs, you can save two frames worth of latency in the best-case scenario.''

You assume everyone plays on last gen consoles and TVs from the 90's. Some have 144-240-360hz monitors with 1ms response time. Redo the math for top of the line hardware console or PC players and come back with a reasonable excuse instead of catering to just casual last gen console players living in the amazon, arctic or deserts playing on Starlink wifi when you have a growing competitive e sports scene.

This is why, throughout this blog we’ve shared a number of improvements that we’re pursuing in the near future, including:

• Using real-time alerting that will allow us to identify problems and respond more quickly
• Implementing tools for identifying servers so we can remove and replace problematic servers rapidly
• Focusing on slow-mo servers—removing problematic servers is one step, but our goal is to make this drastically less common with code changes
• Reducing latency with better optimization of new features
• Fixing hit-reg bugs and building automated detection tools to help us avoid introducing new ones

Reducing latency and better optimisations is something you would have been working on since release. You mention no real improvements coming except the removal of the odd slow server and not how your game runs poorly in general.

You also forgot to mention how SBMM would increase yours, or other players in your matches ping, putting me in eastern european servers (have london selected) who also have players from South Africa. Wonder why i get shot through doors...

CLOSING THOUGHTS

You make excuses for a one size fits all matchmaking for inclusivity that you conveniently left out and compare your improvements to consoles as the investment will cost you money which a lot of PC and next gen console players would see tangible benefits from but because the majority of your players wouldn't benefit or notice, it's not worth the extra cost to you.

@DoYaSeeMe ''Other modes - this divides the playerbase. They know this first hand, from Titanfall 2, that's why they avidly avoid it.''

We have a new 3v3 mode now. They could easily increase it for that or would 6 players be to much for the servers because of bullet physics? Yes it would take more processing on their end but it's technically possible when snapshots are of a small map with 6 players...

 @apostolateofDOOM 

Thanks for this post. I hope this gets more views since season 9 launch was a bit harsh. I agree that there must be something that can be done to improve overall situation how networking is working in APEX.

@DoYaSeeMe 

Thanks for your replies and comments. Earlier on you did write possible solutions and race car analogy. To me these ones were great since they pulled me towards understanding what situation might be on the other end. Dunning-Kruger =).


I dare to claim that improving game general stability and errors would really help in the long run. They have made it. Yes. There still are some issues that are from season 1. I personally would be happy to play with same legends and everything for at least half a year to make most of the occuring bugs to go away (missing sound, code.net, no reg, packet loss).

In the end keep up gaming and thumbs up for Apex Legends.

 See you in arenas..when servers are up. =)

@Littumulkero Thank you.

I'll be updating this thread soon with some feedback to their server post and highlighting where some actual problems are and what most definitely can be improved upon. I'll even do some number crunching for them. 😉

Quick bump to remind myself this thread still exists as do all the other problems mentioned that haven't been addressed.

Will get round to it when i get some free time.