I found these annoying too.
It felt like there was some 'special' way to press the buttons...which I mostly didn't do.
On one hard decyrption, I was flying along getting them right and then...I failed.
It really feels like the game is cheating you by saying "You didn't say 'Simon says' when you pressed that last button...FAIL!".
Not a fan of this 'game'.
