EA Staff (Retired)Thanks for these logs. I'm the engineer who implemented the crash log system. I'm also the one who used the logs when we first implemented the voluntary crash reporting system to find and fix the bug causing the vast majority of the crashes we were seeing. After that fix, we saw the rates of players vanishing from servers plummet.
I just found this thread and am now looking into each of these logs. I see four different crash reports in this thread.
One of them is the one I just commented on, in msvcrt.dll when quitting. As I said, I don't think we can do anything about that one, since it's Microsoft's code that is crashing, but at least it only seems to happen when you already wanted to exit anyway.
The other three are in r5apex at three different offsets: 00000000002F2DCA, 00000000002F2E2A, 00000000002F2E9E. I'll comment on these in reverse order.
The last one (00000000002F2E9E) happened once for TlElTlRlIlS and is the most confusing. That's executing code in the middle of an instruction. That shouldn't be possible.
The middle one (00000000002F2E2A) happened for two people, kaolinAngel_TTV (original poster) and umarusann. This looks like it might be a legitimate stack corruption bug, but it might also be an overclocking / overheating bug. I'm in the process of analyzing the code to figure out how this might be possible. This is hard, and may take some time. I can't just analyze the human-readable code we actually wrote; I also have to analyze the machine-readable code the compiler generated from what we wrote.
The first one (00000000002F2DCA) happened for 12 people in this thread, making it the most common. This one is also bizarre. It's executing an instruction that just asks if a CPU register is zero. (In very basic terms, a register is a place inside the CPU that holds a number that's being used in its computations. In one sense, it's like a super tiny cache of super fast memory. If that doesn't make sense, don't worry about it!) That instruction is causing either a CPU breakpoint to be hit (10x), or causing a write access violation (2x). This instruction should not be able to do either of these things. It's not a breakpoint, it's a bit test. It doesn't write memory, it reads a register.
Interestingly, all three of these crashes are in the same function. This is also the function that sometimes reports an execution access violation on a memory location that isn't the current or next instruction, which should also be impossible for a properly functioning CPU. When that crash happens, I made the latest patch of Apex tell you that the crash is probably due to overclocking or overheating.
So, the most likely explanation I can come up with for these other crashes is that your CPU is overheating, possibly due to overclocking. xjungleWarrior even helpfully mentioned that he had a 3.7 GHz CPU overclocked to 4.9 GHz. Thanks for that extra tidbit!
I have worked as a software engineer in the games industry for nearly 20 years, but before that I earned a Master's in electrical engineering from Stanford, with a specialization in microprocessor design. So, while not an expert on overclocking/overheating behavior of modern Intel CPUs, I do have a lot of relevant experience and background. I'm a very logical and scientific guy. So, I don't suggest overclocking/overheating lightly. I suggest it because it's the most logical explanation.
Overclocking / overheating can cause very rare CPU faults. These can even show up only when you execute a particular sequence of instructions, so you can think that your computer is perfectly stable when it's overclocked, and then one day you install a new program and it crashes a lot. It's only natural to think that it's the program's fault, since everything else worked fine for so long. But CPU clock speeds are set to make sure that the CPU *always* finishes every possible instruction sequence in time for the next instruction to start. If you increase the clock speed, you give it less time to finish instructions. That will usually work, since there will usually be some "buffer" time at the end of each instruction. But in very rare cases, it needed the whole clock cycle to finish this particular instruction with that particular data, and something else need the results right away, and then your CPU will crash.
Overclocking can also lead to overheating, but overheating can happen even without overclocking, such as in a hot room. These can have similar effects as overclocking, but overheating can be even worse. It can lead to more thermal noise inside the CPU, especially when the CPU is working hard, which can cause bits to flip inside the CPU when they shouldn't. If that happens, your CPU is likely to crash.
All that to say, the most likely explanation for the vast majority of these crashes is overheating and/or overclocking.
If you're experiencing these issues, please try these things and reply to this thread. I really want Apex to be stable for all our players, which is why I implemented the crash log in the first place!
Thanks again for all these logs, and hopefully together we'll be able to figure out how to keep you all in the arena!
Please check the code for some sort of "CPU Parity error" corruption, if that is possible.
I do know that one reason for these crashes, which I have managed to log myself, is a "correctable" (meaning: not blue screen) WHEA error, called "CPU Internal Parity Error". This seems to happen randomly and happens usually on overclocked systems but if it is happening at STOCK (that same R5Apex: 00000000002F2DCA) error, then it means there is something seriously wrong. Intel designs their CPU's to run at stock up to 100C (at the highest official turbo frequency). This is absolutely *NOT* an overheating bug at all. These errors can happen even as low as 45-50C. In most people's cases, reverting to pure stock settings will prevent this.
I do know that this "Parity error" was first reported after some old microcode updates (originally for Haswell processors) for Spectre/Meltdown mitigation, although the OS had to be aware for these protections to actually work. What I believe is that these crashes are caused purely by AVX code doing some..."strange" things that I've never seen before.
In my case, I know that programs that use AVX and hit the L2 cache heavily, like Realbench 2.56 and Cinebench R20 (new version) start going pear-shaped in an overclocked system at about 94 degrees celsius. However the errors generated here are (CPU Cache L0 errors) (correctable WHEA Errors), and if it is not correctable, a System service exception, Kmode exception not handled, or IRQL Not Less or Equal BSOD will be generated. (funny enough it's never Clock Watchdog Timeout (stop 0x101--frozen cores) or WHEA uncorrectable error (stop 0x124--usually a cache error). A CPU Cache L0 error isn't even a cache error relating to normal CPU caches--it's some sort of instruction register (CPU's don't have a L0 cache). There have been some comments that this is an "Operand Cache" or Register file, or access to the stack.
These L0 errors are the "overheating" errors you are referring to. You only get these errors when you are overclocked and your CPU exceeds a certain temperature in a stress test or full usage application.
Apex isn't generating these errors. It's generating "Internal Parity Errors". These are most likely causing the game to crash. But these errors are happening at VERY Low temps.
This could also be some sort of Intel Microcode bug as well. But I have no way at all to check on such a thing. That being said, the newest 9900K microcode you can download and patch (in windows, without bios flashing) yourself is ucode AE, which doesn't do anything helpful for this.
Please check on this however. This "CPU Internal Parity Error".
But for those who are getting these crashes at *STOCK* speeds, this is not an overheating issue and should not be happening. There may be a bug with the AVX code or something Apex legends is doing.
Community Manager
