Re: Apex Legends Crash no error - PC (apex_crash.txt)

@MrDakk @OrioStorm 

Please keep me posted.

And you were right.  I had three hours of stability, but then suddenly the "internal parity errors" started happening again, and not long after, the random crashes.

The only fix was to increase CPU voltage -excessively- high.

Oriostorm, is Apex doing something timing related that is causing this instability?

Because it really should not be happening.

I set my cpu to 5.2 ghz (hyperthreading off) at 1.335v and I ran Battlefield 5 (Firestorm) for an hour.  BF5 was using >75% of the CPU cores, temps and power draw were much higher, and there were no errors or crashes.  I then ran the Blender Classroom render stress test (you can google this), and blender BMW stress test and both completed with flying colors.  The only programs that had problems at this point were Apex Legends (those crashes and Internal Parity Error) and Prime95 AVX 1344K (clock watchdog timeout BSOD).  Prime95 small FFT with AVX disabled ran forever.  (Prime95 29.8 build 1).

Increasing CPU voltage to 1.385v completely stopped the Prime95 1344K AVX tests from crashing. These ran fine now.

But APEX legends was still randomly acting up (exception error, or CTD sometimes with no error, or internal parity error (no crash).

1.395v-1.40v was needed to stop this--far far beyond what any other program or application needed!  Something is really strange here.

And what's even worse---Apex Legends uses FAR FAR far less CPU resources than Blender, Battlefield 5, Prime95 (AVX disabled), etc.

Oriostorm, is there any way, if you have time, to write a very small 'stress test' code sample that can extensively test some of the instructions you are using for Apex Legends, in a repeated intense loop of cycles, so that we can run the executable program and it can catch any SSE2 error in a bugtrap?  This may help track down the problem.  This might also help determine why some users with stock clocks are also getting these crashes (in most cases, pure stock clocks prevent these problems).  It shouldn't take that long to write a code sample that we can download, and it may help determine what's going on.

It really is inconceivable that Apex is needing far more voltage than something like Blender or Prime95....

Could EasyAntiCheat be causing these timing errors?

By the way, this "bizarre" behavior seems to get reversed (meaning things start acting properly) if you run at base clocks (4.7 ghz) and then downvolt the CPU enough so it's unstable.

Doing this and Apex Legends runs nice and happily while Prime95 with AVX disabled generates a BSOD crash.  This happened at about 4.7 ghz (HT enabled) at 1.065v. 

You have to go even lower on the voltage before Apex starts generating parity errors.

@Falkentyne, I honestly don't know how Easy AntiCheat works (I wasn't involved in integrating that), so I can't speak to whether it's a factor.

I do think it's related to the actual instruction sequence, and possibly it's offset in memory. I can try making a standalone test program, that's a good idea. But it's not as trivial as you might think. This function does connect with the rest of our engine, so we have to sever those connections to make a standalone program, but we have to do it in a way that doesn't change the generated assembly.

If that doesn't work, I guess I could write an assembly language file that just has this function exactly as it is in the live game, and try to have external code feed it the data expects in a way that won't crash.

Another tricky thing is that the path through this code depends on branches based on what you can see in the game. If a particular path through the code is causing the bug, you'd need to replicate the live data that causes the crash. That data can be different for every viewpoint in King's Canyon, so even replicating all of the code may not replicate the crash if we don't replicate the control flow decisions. There are things we can try to force different control flows, but it's not guaranteed to repro.

Still, it's worth a shot!

@OrioStorm 

Thanks for doing this. I've got 3 crash logs for you; the first two are from patch 1.1.0 and the third is just last night from 1.1.1. I've got about 20 more if you want to see them - hex error codes are 2F2BB0, 2F20B6, 2F239A, 2F20C1, 2F23C5. The latest ones are 2F2E2A and two that have 2F2DCA.

History of my crashes - I picked up the game in the first week of release and didn't have a single crash until the patch that fixed the revive bug - The bug where a revived character still moved super slowly as if they were still down and wounded. After that patch, I started crashing anywhere from a couple times every few days to eleven times in one day. It was random, and it was super frustrating, but I kept going regardless because I really liked the game. The crashes came at all times - most were in-game, but a couple of them happened while I was still in the character select screen. None of them seemed to have anything in common with each other - I crashed in gunfights, I crashed in peaceful fields, I crashed with apps like Discord open, I crashed with just Apex open, and a dozen other random situations it would take too long to recount here.

But with the 1.1.0 patch, everything went to hell. I crash every single game now, every single time, usually within less than five minutes. I don't know what changed, but whatever my problem is, the 1.1.0 patch kicked it into overdrive. I can't play at all now, haven't touched the game at all since the day 1.1.0 dropped aside from last night so I could generate a fresh crash log for you. Crashed about 3 minutes into the game, just like I knew I would. I hope you manage to figure out what's going on; I still like Apex, but the game has made it clear it doesn't want me playing it.

System information - Intel i7-9700k overclocked to 4.9 gHZ, voltage at 1.235

Ram - 16 GB overclocked to 3200

Graphics Card - Nvidia 1080, stock timings

Types of crashes - I get three different types of crashes, detailed below:

The typical crash - The visuals freeze, audio may or may not keep going, and after about 2 - 4 seconds the game crashes to desktop. This crash hits me about 45% of the time and DOES GENERATE A CRASH LOG.

The atypical crash - The visuals freeze, audio may or may not keep going, and after about 2 - 4 seconds the game crashes to desktop. This crash hits me about 50% of the time and DOES NOT GENERATE A CRASH LOG. I have no idea why this crash doesn't generate a log. I assume something is breaking that isn't being accounted for in your logging process, or Windows or possibly Easy Anti-Cheat is somehow preventing log generation. If I had a log for every crash, I'd have somewhere north of 50 by now.

The "Instant" crash - One second I'm in the game. The next second I'm staring at my desktop. No transition whatsoever between the two - it's that fast. No audio or visual freezing, no screen tear, just instant transition to desktop with the game mysteriously closed. Obviously, THIS CRASH DOES NOT GENERATE A CRASH LOG. It only happens about 5% of the time, but it sure makes a hell of an impression when it does.

Finally, one crash I had that might help you - don't think it generated a log: One time, about 3 weeks ago, so before 1.1.0, I had a situation in-game where I thought I crashed - The screen froze, audio kept going, and I thought welp, there goes this game. But then, for some mysterious reason, after about three seconds the game snapped back and resumed normal play. I could run around, shoot, hear everything normally for about five seconds. THEN the game did an "Instant" crash - straight to desktop with no transition at all. I have no idea what it meant - perhaps the function somehow managed to correct itself briefly? Or some other code process interceded? But it was weird enough that it stuck out and made me want to mention it.

Hope all this information helps you. I really want to play Apex without wondering when the game is going to kick me out of it.

@OrioStorm Thank you very much.  I would be happy to do these tests for you (after all it's what we people over on overclock.net do, spend all day running stability tests!).

I just did something very interesting on my laptop.  

It is a 7820HK (MSI) laptop, and I set it to a speed which I know is unstable:

4800 mhz (cache: 4400 mhz), and 1.260v.

This laptop does not have loadline calibration so there is voltage vdroop, but there is no "live" Vcore sensor, just VID, so real vdroop is impossible to measure.

Anyway:

This laptop is highly overclocked to its max absolute limits.

At 4800 mhz, 1.260v, Apex Legends generated three CPU errors but did not crash:

1 x Internal Parity Error on thread #4

2x Cache Hierarchy Errors on threads #6 and #3. (the threads go from 0 to 7, one physical core+1 logical core, so threads 0 and 1 are for the first core, etc).

This was a 1 hour play session.

This is exactly what I would expect an unstable CPU to do.  

These cache hierarchy errors are also what Realbench 2.56 and Prime95 (AVX disabled) do on my desktop when they are unstable.  So this is properly expected.

Apex Legends did NOT crash however!  But the fact that I got *both* internal parity and Cache L0 errors (instruction register corruption basically)--but they were corrected, means that the "Unstable" CPU is actually acting as expected.

1.265v did not generate anything but I didn't test that long enough.

The 9900K desktop, when overclocked is performing completely different however, although desktops do have a loadline calibration setting to lower CPU vdroop.

Instead of throwing any L0 cache errors in Apex, it just throws out Internal Parity Errors, or just exception error crashes Apex, while every other stress test or program (like more intensive stuff like Battlefield 5, Blender, etc) runs all day (except small FFT AVX prime95. which is a 200 amp 100C temp power virus; 1344K AVX prime is more realistic).  Small FFT prime with AVX disabled also runs fine.

My gut is screaming "Microcode bug" but unfortunately, disabling loadline calibration on the desktop wouldn't work too well. as then you would need unsafe voltages to be stable with <>200mv of voltage droop.  I may try disabling 4 cores (e.g. turn my 9900K into a 7700K) and then overclock and disable LLC (Loadline calibration) later, and test, but this is very time consuming.  

I would enjoy doing any small code testing for you.  If you have a more convenient way I can contact you or where you can post anything to test, let me know.  I'm always available.

I’d like to add that all of these crash types you’ve explained are happening almost identically for me. I get all three types of those crashes. I have yet to see the 2F2DCA crash since the newest patch, though I haven’t played a whole lot since the patch dropped. I did crash multiple times with and without crash logs and posted a log from the new patch. The crashes I’ve got are the EXCEPTION_ACCESS_VIOLATION(write) and EXCEPTION_ACCESS_VIOLATION(read). I don’t have the error codes handy at the moment. I’ll definitely post more logs as I get them. 

Also want to thank @OrioStorm for all the input and support on these crashes. It’s rare to get this kind of insight and help on forums nowadays from devs. So thank you for taking time out of your day to respond, educate, and help solve these issues. Thanks to everyone else as well who’s continuing to test and provide information. I really hope a solution is found soon. Keep providing as much info as you can.

I strongly advise everyone in here to revert their overclocking CPU settings back to default,

and disabling Intel SpeedstepTM in their BIOS.

I am pretty sure this has solved my problem as no longer do I have these crashes with logs.

Before it used to be all the time, now I can go for days without a single crash.

As I read here, since the problem was CPU related I tried different things.

But as mentioned over, im 99% this has solved my problem completely.

im using 7700k and 1080 ti, if any others at least try stock clock speeds and disabling Intel Speedstep,

it would be very interesting to see if the crashes are actually stopping.

@TEZZ0FIN0 

My crashing stopped as well, I'm using an i9-9900K. I was stable at 4.7GHz (Turbo Boost) so I just introduced overclocking back and went to 4.8GHz to see if I'm stable. I'll report back soon.

crash:
{
atidxx64: 0000000000760244
EXCEPTION_ACCESS_VIOLATION(unknown): 0000000000000000
}
cpu: "AMD FX(tm)-8350 Eight-Core Processor "
ram: 16 // GB
callstack:
{
kernel32: 000000000009B9B0
ntdll: 0000000000079015
ntdll: 0000000000057388
ntdll: 000000000006BF7D
ntdll: 000000000004043A
ntdll: 000000000006B61E
atidxx64: 0000000000760244
atidxx64: 0000000000032617
atidxx64: 000000000072DAAF
atidxx64: 0000000000722239
atidxx64: 00000000008BBFB4
kernel32: 000000000001556D
ntdll: 000000000005385D
}
registers:
{
rax = 16
rbx = 34
rcx = 0x62A86B40
rdx = 2
rsp = 0x4204FB20
rbp = 0xB4BA0B12B50D3CF6
rsi = 0x62A86B40
rdi = 0x339F6ED0
r8 = 4
r9 = 0x339F6ED0
r10 = 0x62A86B40
r11 = 0x4204FB60
r12 = 0
r13 = 0
r14 = 2
r15 = 4
rip = 0x000007FEF3770244
xmm0 = [ [-2.0458958e-43, 9.6272987e-35, -3.9290089e+24, 0], [0x80000092, 0x06FFF000, 0xE8500000, 0x00000000] ]
xmm1 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm2 = [ [0, -nan, -nan, -nan], [0, -1, -1, -1] ]
xmm3 = [ [120.36296, 0, 0, 0], [0x42F0B9D6, 0x00000000, 0x00000000, 0x00000000] ]
xmm4 = [ [270, 0, 0, 0], [0x43870000, 0x00000000, 0x00000000, 0x00000000] ]
xmm5 = [ [67.26667, 0, 0, 0], [0x42868889, 0x00000000, 0x00000000, 0x00000000] ]
xmm6 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm7 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm8 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm9 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm10 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm11 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm12 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm13 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm14 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm15 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
}
build_id: 1560910906

Crash log.