Re: AT&T network - "Unable to load persistence data" - Season 5

Hi all, 

Please read the following carefully.

For the players still experiencing this issue - and with AT&T as their ISP -, could you please perform a diagnostics via your AT&T router and DM me a screenshot of the results?

!!WHEN SHARING YOUR DIAGNOSTICS RESULTS, PLEASE ALSO MENTION YOUR OWN PHYSICAL IP ADDRESS (which you can find via whatismyipaddress.com)!!

You can run the diagnostics this way (you need a PC for this that is on the same network as the network you're playing on):

Via your AT&T-provided router admin panel:

• Open a web browser and go to http://192.168.1.254/,
• Login to the admin panel of your router,
• Go to the Diagnostics tab > Troubleshoot,
• Where it says "Test Internet Access", enter the following and click on Traceroute:

kingston-prod-wgw-envoy.ops.dice.se

• Let it finish all hops & take a screenshot,
• Please send me a screenshot of the results via a DM here on AHQ.

Here you can see where you can find the Diagnostics > Troubleshoot tab in the admin panel of your router:

Via your Windows Command Prompt (CMD)

• Open your Windows Command Prompt by going to Windows Start Search and typing "cmd",
• Enter the following and press enter: 

tracert kingston-prod-wgw-envoy.ops.dice.se

• Let it finish all 30 hops & take a screenshot,
• Please send me a screenshot of the results via a DM here on AHQ.

*While the results indeed show some IPs, this information is only being used as troubleshooting for AT&T.

Thanks in advance.

what are we looking for that tells us something isnt right?

I have a screenshot of our AT&T test bu I am unable to send it to you via DM. Please DM me and I can send to you. It has been working for a couple of days but now getting the same persistent data error. My husband and I are on the same network and he can get it in, but I can't. He is playing on PS5 and I am on PS4. 

@nola_rosI was also not able to PM until I created my first forum post. You should be able to private message him now. Just click on his profile and you should see an envelope next to his name. Click on that and you should be good to go!

Thanks!

Error code attached. Cannot connect as of 12:01AM EST

traceroute to kingston-prod-wgw-envoy.ops.dice.se (52.215.43.89), 30 hops max, 38 byte packets
1 23-115-64-1.lightspeed.knvltn.sbcglobal.net (23.115.64.1) 1.774 ms 0.739 ms 0.821 ms
2 99.17.140.210 (99.17.140.210) 0.892 ms 1.101 ms 0.885 ms
3 * * *
4 * * *
5 * * *
6 32.130.16.203 (32.130.16.203) 13.392 ms 11.572 ms 12.419 ms
7 192.205.33.42 (192.205.33.42) 11.933 ms 11.722 ms 9.796 ms
8 atl-bb2-link.ip.twelve99.net (62.115.143.236) 9.288 ms 10.857 ms 10.894 ms
9 ash-bb2-link.ip.twelve99.net (62.115.137.132) 23.477 ms prs-bb1-link.ip.twelve99.net (62.115.112.243) 103.881 ms rest-bb1-link.ip.twelve99.net (62.115.138.70) 29.394 ms
10 prs-bb1-link.ip.twelve99.net (62.115.112.243) 107.825 ms prs-b3-link.ip.twelve99.net (62.115.118.59) 106.131 ms prs-bb1-link.ip.twelve99.net (62.115.112.243) 106.075 ms
11 * prs-b3-link.ip.twelve99.net (62.115.118.59) 106.066 ms 108.144 ms
12 * amazon-ic-367021.ip.twelve99-cust.net (62.115.42.69) 105.481 ms 105.608 ms
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

Please do NOT post in this thread if you do NOT have AT&T as your ISP.

If you have a different ISP, please post in this thread.

Thank you.

Update Aug. 2, '23:

Big thanks for providing us with all the examples/reports! You're all truly amazing. We'll be sharing these reports with AT&T & AWS in hopes they can pinpoint the root cause of the issue.

For now, we don't require any new diagnostic results. If that changes, rest assured, we'll keep you informed.

And if there are any updates to share with you, we'll make sure to do so promptly!

- EA_Leeuw

Edit:

Just wanted to add that while the Battlefield Comms Twitter channel just released this tweet, this is not related to the AT&T issue that this thread is about. 

EA_Leeuw, Im serious man. Should start giving out battle pass refunds asap. Bought this game maybe one week ago and EA has allowed me to play 3 out of those 7 days.

Just confirming this was fixed for me a few days ago, but is no longer working now.

Taking a look at this again, and as far as I can tell the issue is still roughly the same - tons of TCP retrans/TLS issues (likely from TCP segment loss) when connecting to: pub-bf-prod-grpc.ops.dice.se (application actually looks up kingston-prod-cgw.ops.dice.se which has a CNAME record that points to pub-bf-prod-grpc.ops.dice.se)

It seems like EA did make some changes here.

Old IPs:

108.128.42.217

34.248.44.177
52.208.207.227

Current IPs:

108.128.42.217
99.80.71.123
54.74.237.202

Previous behavior before the "fix" was only traffic destined to 52.208.207.227 was problematic. Now all 3 current IPs are problematic and show the same errors in the PCAP. Not sure how it was temporarily working, but the issue still seems scoped to that service.

Very bizarre that it seems time of day related as well.. something is on a schedule.. not familiar with service provider networks but I can't think of much that would be on a schedule to cause this. QoS/Traffic shaping based on time of day? policy based forwarding/routing somewhere?  To affect this many users if it's not a core routing problem (which if it was it would have to be fixed by now, no way it would take this long) it must be taking a common path with a problematic device somewhere. Makes it seem more on the AWS side (EA config or infrastructure who knows).

Would love to hear the engineer's current thoughts on this - seems almost unbelievable that it hasn't been resolved yet so either there's some incompetence at play or it's a very interesting issue.

---

@EA_Leeuw wrote:

Update Aug. 2, '23:

Big thanks for providing us with all the examples/reports! You're all truly amazing. We'll be sharing these reports with AT&T & AWS in hopes they can pinpoint the root cause of the issue.

For now, we don't require any new diagnostic results. If that changes, rest assured, we'll keep you informed.

And if there are any updates to share with you, we'll make sure to do so promptly!

- EA_Leeuw

Edit:

Just wanted to add that while the Battlefield Comms Twitter channel just released this tweet, this is not related to the AT&T issue that this thread is about. 

---

Update Aug. 3, '23:

Just wanted to share the following Battlefield Comms tweet.

Please keep in mind that this thread is about the "Unable to load persistence data" error, with AT&T as your ISP, only.

For general "Unable to load persistence data" - Season 5 errors and troubleshooting, please take a look at this thread.

The two issues behave separately. The AT&T issue results in the "Unable to load persistence data" error message while you're booting the game, on the "Connecting to servers" screen, not while trying to matchmake into a specific mode or playlist.

I was able to play yesterday for awhile around 5pm EST via ATT Fiber. Later I switched to another game and came back to BF later only to get the error.

Just wanted to chime in here to say this is absolutely unacceptable for a AAA game.  I can play on PC throughout the day but as soon as I try to log in at what I suspect are high-traffic times, (9pm EST) I get the persistence data error.  

I shouldn't have to use a VPN to play the game.  I've never had this issue with any other title in 20+ years of gaming. 

EA:  take responsibility and do better.

Hi @CiggyTardust,

Thank you for sharing your feedback. We truly understand that it's frustrating.

Have you seen the updates of this thread that are being shared?
I would like to highlight the accepted solution of this thread.

Not seeing any proposed solution?  If you mean restart my modem/computer I do that every day.

@CiggyTardust, it's on the 1st page of this thread. I linked it in my previous post. 🙂 The post is called an "Accepted Solution", it's also stated in the post why it's marked as a solution.

Hope this helps.

Ok so there's no solution.  Still being investigated.  That's quite a solution.  Got it.  

That EA has been this careless about fixing an issue that’s been preventing a large number of players (ATT largest ISP in US) from playing for 2+ weeks is absolutely unacceptable. What a slap in the face to fans who have stuck with this franchise for so long. Guess it’s time to move on to other games that actually allow their players to enjoy what they paid for. 

---

@GroovyDoovy wrote:
@5ubj3ctz3r0I wonder if it's not time of day related but volume of traffic related, and that maybe the volume of traffic tends to start going up around 7pm CST?

---

Can't speak for console/Origin/etc users - but for Steam users you can see some historical graphs using steamdb.info ( https://steamdb.info/app/1517290/charts/ ) and there is actually a sharp drop in player count around that time, consistently too (could be AT&T users).

Checking again this afternoon when it does connect and there is no change in which IP addresses I'm connecting to - it's the same ones except during this time there is no errors. If it was bandwidth or load related I would assume that the experience would be intermittent not consistently unable to log in during those times.

AWS, and AWS customers often have redundant designs for their applications - makes me wonder if at a certain time whatever sits behind those before mentioned IPs does some kind of routine maintenance and fails over to other "servers" that are problematic. Whether that's AWS underlying infrastructure causing dropped packets somewhere (though this is AT&T routes specifically, so this seems unlikely), or an EA configuration difference between "A" and "B" sets of "servers" that sit behind those IPs who knows.

I took a trace route when it's working to see if it's different when it's not (I don't expect there to be a change)

Working:

C:\Users\M>tracert -d 99.80.71.123

Tracing route to 99.80.71.123 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 10.3.5.1
2 1 ms <1 ms 1 ms 10.5.5.1
3 1 ms 1 ms 1 ms 108.232.84.1
4 1 ms 1 ms 1 ms 71.151.17.114
5 8 ms 7 ms 7 ms 12.242.114.16
6 22 ms 22 ms 29 ms 173.241.128.29
7 108 ms 136 ms 99 ms 89.149.185.49
8 142 ms 142 ms 143 ms 154.14.130.134

EDIT:

Not working:

C:\Users\M>tracert -d 99.80.71.123

Tracing route to 99.80.71.123 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 10.3.5.1
2 <1 ms 1 ms <1 ms 10.5.5.1
3 1 ms 1 ms 1 ms 108.232.84.1
4 2 ms 1 ms 2 ms 71.151.17.114
5 9 ms 7 ms 6 ms 12.242.114.16
6 3 ms 4 ms 2 ms 173.241.128.29
7 89 ms 89 ms 90 ms 89.149.185.49
8 151 ms * 143 ms 154.14.130.134
9 * * * Request timed out.
10 ^C
C:\Users\M>time
The current time is: 20:15:35.92

Same path as expected - same previously noted behavior. Appears to happen at 8PM CST.

(Note it doesn't actually make it to it's destination but that just means that ICMP is being dropped beyond that last hop, doesn't mean it's unreachable.)

Comedy 

For what it is worth I see alot of TCP port 0 happening.  I have the pcap from my firewall if someone at EA would like to view it.   Also the reason the hosts file mitigation doesn't work anymore is because it seems the Battfield Client completely ignores it.  They must have their own mappings in the program.   

P.S. If this issue keeps going on without being solved I predict it will finally kill the game. I already played a game earlier today with nothing but AI. Just saying...