EA Staff (Retired)I want to thank everybody in this thread for their help on this issue, especially @Falkentyne, @JorPorCorTTV, @MrDakk, and @TEZZ0FIN0.
Based on all the logs and investigation we've done, I'm now convinced that this is a flaw with Intel chips. There is some sequence of instructions that causes results to be used before they're ready on Intel chips. There is one function in Apex that executes this sequence. It doesn't have to be a complex, CPU-intensive program to expose flaws like this; they could even show up in Notepad! I know that we have other functions that are heavily optimized that don't crash; this function that crashes so much is hardly optimized because it's so simple.
It seems that overclocking Intel chips can cause this to happen pretty reliably. It also seems that this can happen when Intel SpeedStep technology raises the clock speed without actually overclocking at all. In all cases, it seems that lowering your clock speed causes the crashes in this code to stop.
So, I base my conclusion that it is an Intel hardware flaw on the experimental evidence that ⚽ the information about the CPU state that the OS reports for these crashes is always impossible for a properly functioning CPU, 🏈 these crashes are only reported on Intel chips, and 🏀 lowering the clock speed always fixes these crashes (even if it wasn't overclocked).
These crashes are exceedingly rare from a CPU perspective. If you crash every other game, that feels like a lot, because it is. My personal goal is nobody crashing ever! However, say that crashing every other game translates into crashing about once every 20 minutes. The CPU runs the instructions that crash at least 100,000 times a second when you're playing the game. With these conservative estimates, the CPU crashes about once every 120 million times it runs this code. So, even though it truly does crash a lot, the conservative estimate is that even a malfunctioning CPU actually functions properly in this code about 99.999999% of the time.
So, what next?
Well, I tried to isolate the crashing function to make a standalone program to exhibit the CPU bug. Unfortunately, I couldn't get the compiler to generate the exact same disassembly. Since the crash appears to depend on the actual sequence of instructions the CPU actually runs, if the disassembly is not equivalent, I don't think it's a good test. Even if I had identical instructions, I don't know that I could reproduce a data set that would cause this function to crash. The real data set depends on where you are in King's Canyon and which way you are looking. I can't replicate all the code and data to generate this data set in a standalone test program, so I'd have to generate random data and hope it crashes. But we've never seen this crash on any of our work machines, so I don't really have any way to verify that I've come up with a crashing data set.
All this to say, I've decided that it takes too much time to make a standalone program to try to repro this bug. The program has a low chance of succeeding, and I can't locally test whether it succeeds or not. Even if it we got lucky and it happened to work, then it would only help highly technical people hone-in their clock speeds, and it might help Intel fix their hardware flaw.
But, it wouldn't help everybody else who has an Intel chip who keeps crashing and doesn't post here and who doesn't want to mess with their clock speeds. I want to help them too! And as a consumer, I know that when you're crashing you don't really care whether it's the CPU's fault or the video card driver's fault or the game's fault; you just want the game to work.
So, I'm going to try changing the function that has all these impossible crashes to do the same job in a slightly different way, and hope that nudges it out of the "sweet spot" that causes some Intel chips to crash every few matches. Until we verify the fix, I'll leave in the old way as a hidden option, so we can be sure that we don't accidentally make things worse with no quick way to go back to the way things were. This is still a shot in the dark, because we can't repro the crashes on any of our machines, but it's the best shot I can take based on all the evidence in this thread.
Unfortunately, I don't know when these changes will go live through our regular release schedule.
@OrioStorm Thank you very much!
I knew it had to be a bug. I guessed (if you look at one of my posts earlier) i mentioned some sort of internal bug.
I guessed this because you remember, I mentioned "internal parity error" right?
That error is the "bug" happening and then being corrected by Error correction (ECC).
If you decrease the voltage even lower, you get a "Translation Lookaside buffer" error (instead of just internal parity errors), so these bugs are happening somewhere possibly in the TLB area (all CPU's have them), in an instruction register, but NOT in the "L0" cache (L0 cache is basically some sort of register also, almost like a bridge between the cores and L1 cache).
The only way we can fix this is to contact Intel.
They will need to release YET ANOTHER microcode patch which can fix this bug.
Can you please document your findings, if possible, and see if you can either contact Intel, call their 800 number, or post on one of the Linux / Debian developer threads so that this bug can be sent to Intel?
This is NOT in any way similar to the Pentium "DVID" bug F00F bug, as that was an instant 100% guaranteed crash, or maybe it's more similar to that Skylake FFT bug, where certain prime number FFT sizes would crash the processor. (this was fixed in a microcode update).
I'm just a gamer so I have no access to Intel. But since you are a developer, you may be able to reach them. Reference the crash threads on these forums and hopefully this will be escalated to a "high priority" bug, since SOME users with stock CPU's are encountering this also.
These are the only links I can give you to help get this addressed by Intel
Their tollfree tech number of course (you will have to somehow reach their programming department, good luck with that)
https://www.intel.com/content/www/us/en/support/contact-support.html
(Intel):408 765-8080
(and some 800 number but there seems to be a business to business relations link only)
https://github.com/platomav/CPUMicrocodes
https://downloadcenter.intel.com/download/28727/Linux-Processor-Microcode-Data-File
*Edit* a number that works:
(916) 377-7000
@OrioStorm I know it's a post bump, but I contacted Intel on the phone, talked to some technical guy and explained that there may be an internal flaw or microcode bug in the 7600k-9900K processors (not sure about older Haswell or HEDT), and referenced your posts and the other thread in a followup email reply.
Let's hope they identify a code sequencing bug or something similar to that Skylake Prime number FFT type bug, can replicate it and it can be fixed in a microcode update.
Hopefully either you or Intel will find some way to establish communication so the processor team has a go at it.
Drop the overclock down and it should stop crashing. Was running stable at 5GHz and never had issues with anything, but with Apex crashing all the time with the same error you stated. Dropped mine to 4.4GHz and never crashed. I brought it back up to 4.8GHz and I have yet to crash still and it’s been roughly 25 hours of gameplay. I reset my bios completely so all settings were default. I disabled EIST and Turbo Boost. Then set the fixed OC and left everything else default this time around. Have not had an issue since. It’s an overclock issue with Intel CPUs and some instructions that Apex runs. Sucks, but man I’m happy I haven’t crashed at all and really only sacrificed 0.2GHz. I can live with that.
I can understand that, but with everything else I do on my computer, I will not be dropping my overclock for a free to play game. I lose nothing by not having a great experience except my time. I paid for the cpu, so getting everything out of it means more to me than playing the game itself.
Thats definitely fair. Though if you wanted to play the game, I could see you dropping from 4.9 to 4.8GHz and be fine. This is believed to be an Intel issue. Even though Intel supports overclocking there are far too many settings and changes with overclocking that isn’t covered by Intel otherwise they supply documentation on it. Their CPUs are working perfectly fine under their specifications and therefore I do not see this issue with their high end CPUs being fixed, but we can only hope. I’ve never had a single issue with any piece of software, hardware, or games when at 5GHz and yet something in Apex triggers a crash when Intel CPUs are clocked overly high. Read the rest of this thread if you haven’t yet, one of the programmers for Apex has given a lot of greatness insight on it. Hope this helps.
You can use Throttlestop 8.70 to lower the clocks while in windows. Then when done playing Apex, just set them back up again (you can even save the lower clocks to a profile, and the original clocks to the default profile (#3 IIRC).
Ive read through all of the posts so far in the thread and it's clear that for now, my crashing probably wont be going away until I lower the overclock, and if thats the case, I wont be playing. And thats fine.
Funny thing is, the game ran completely fine until I used Rivatuner to cap the frames and after which is when the crashing occurred. Stopped using rivatuner and crashing stopped for a few days and now its back to doing it every other game or so. Like I said before, some days it plays fine and I have zero issues, some days it's not. Basically a coin flip.
Yea it pretty much sounds the same as the rest of us that are getting this crashing. Like I said before if you still wanna play the game you’ll likely be unable to tell the difference going from 4.9 to 4.8GHz, but that’s your choice. Doesn’t hurt to try it if you’re willing. You don’t have to play the game, and I almost quit myself cause I was sure it wasn’t the overclock causing the issues, but after extensive testing and help from others on the forums it’s definitely the overclock. Haven’t crashed or had a single issue with the game after roughly 25 hours of playing now. At the end of the day it’s your choice man.
I have 8700K OC as well and keeps crashing. I think the problem isn't the OC, it's the game code/latest patch. Before the last patch I didn't crash at all. Revert an OC for one game is inconvenient for me, I'll probably just stop playing.
I play games far more demanding like BF1, R6 and others with no crashes. I play older games and games on Source Engine (that Apex uses) and no crashes as well. And I'm not mentioning my 100+ single player games that I've played with no problems at all. They've just messed something up at the latest patch.
I don't doubt that the overclock is the issue, just hard for me to give up my overclock, or even lower it a tad over something I have zero money invested into when things I do pay for work as intended. Who knows maybe I'll give it a go, but there are other battle royal games out there if I needed to scratch that itch. Ill frequent this thread to see if any news comes out.
im not going to change clocks on my cpu everytime i want to play one specific game ... but using the setting " use cpu offset when running avx" did the same for me
changed my 7700k from 4,9ghz to -4 offset 4,5 standart boost clock and didnt crash since then
Mine crashed w/o OC
crash:
{
msvcrt: 000000000002A9E6
EXCEPTION_ACCESS_VIOLATION(write): 0000000000000000
}
cpu: "Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz"
ram: 64 // GB
callstack:
{
KERNELBASE: 000000000008667C
ntdll: 00000000000A810B
ntdll: 000000000008FD56
ntdll: 00000000000A46AF
ntdll: 0000000000004BEF
ntdll: 00000000000A341E
msvcrt: 000000000002A9E6
sapi_onecore: 000000000011CA8F
sapi_onecore: 000000000011CAC9
sapi_onecore: 000000000003F4D1
sapi_onecore: 000000000015E8C7
sapi_onecore: 0000000000026C30
sapi_onecore: 0000000000026BE4
sapi_onecore: 0000000000026B7B
module@00007FFEA05E0000: 000000000009759D
module@00007FFEA05E0000: 00000000000978CF
module@00007FFEA05E0000: 0000000000097C5C
ntdll: 0000000000048F07
ntdll: 00000000000435BC
ntdll: 0000000000066DCD
KERNEL32: 000000000001D62A
R5Apex: 0000000001241584
R5Apex: 0000000001241520
R5Apex: 00000000004AE602
R5Apex: 0000000001202656
Activation64: 0000000000011050
Activation64: 0000000000008F82
KERNEL32: 0000000000017974
ntdll: 000000000006A271
}
registers:
{
rax = 0
rbx = 0x00000035FBDAEA10
rcx = 0x013FFC2AA8F00000
rdx = 0
rsp = 0x00000035FBDAE9A0
rbp = 0x000001553C7C6F80
rsi = -1 // 0xFFFFFFFF
rdi = 0x000001553C792800
r8 = 0
r9 = 0x00000035FBDADFB0
r10 = 0x00007FFED9D81DC4
r11 = 0x00000035FBDAE3E0
r12 = 0x000001553C7C6F80
r13 = 0
r14 = 0x00000035FBDAEA08
r15 = 0x00007FFEB2E54070
rip = 0x00007FFED7D3A9E6
xmm0 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm1 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm2 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm3 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm4 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm5 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm6 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm7 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm8 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm9 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm10 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm11 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm12 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm13 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm14 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
xmm15 = [ [0, 0, 0, 0], [0, 0, 0, 0] ]
}
build_id: 1554860081
EA Staff (Retired)@Kaz3ee, you didn't get the usual Intel crash. Instead, your program crashed in Microsoft's C runtime DLL (msvcrt.dll) when shutting down Microsoft's speech API DLL (sapi_onecore.dll) when you were exiting the game. I've seen this 3 other times, but I'm not too hopeful that we'll fix it ourselves, since it happens in Microsoft's code when Apex has almost finished exiting. Do you have any info about what you were doing when this happened that might narrow it down, such as quitting while speech was playing?
I was killed and the screen just froze. Wasn't quitting the game.
EA Staff (Retired)@Kaz3ee, that's really interesting. That sounds like a freeze, not a crash, but often the game freezes for a few seconds before finally crashing. Crashes and freezes are basically the same from a user's perspective (the game suddenly and frustratingly stops working), but they are actually different underlying problems.
In a crash, the program did something the CPU doesn't allow, such as trying to read from memory you don't have read access to. Because of that, the OS completely kills the program.
In a freeze, the program is still running, but it's stuck. This comes in two basic kinds. One is the "infinite loop", where the program keeps doing the same thing over and over, but never actually makes any progress so that it can stop doing it. It's basically stuck running in circles rather than running toward the finish line. The other is "deadlock", where thing A is waiting for thing B, but thing B is waiting for thing A, so neither will ever go first and the game just stops. This is basically two guys both wanting to go the finish line, but each insists "no, you go first", so that neither takes a step ever again.
When you get a freeze, you have to kill the program yourself. Apex won't generate an apex_crash.txt for a freeze, because it doesn't know it happened. It only generates the apex_crash.txt for the crash.
So, there are two basic ways to know if this apex_crash.txt goes with this crash or an earlier one:
Thanks for your help!
The one I posted is the crashed after I get killed, and I have to kill the program myself.
The one that I killed someone and I have to terminate the program doesn't gives out any crash log.
