Magic World won't let me speed up time?

As the name says, whenever I enter the realm of magic and want to speed up time, it usually slows me down a few seconds after asking it to do so. However, the speed-up button is still toggled. I can ...