Magic World won't let me speed up time?

As the name says, whenever I enter the realm of magic and want to speed up time, it usually slows me down a few seconds after asking it to do so. However, the speed-up button is still toggled. I can go back to normal speed and back, but the speed slows down a few seconds later again. Unsure on what is causing this, as my character has done several things as this has happened- making potions, dueling, looking for/ reading books... Anyone else having this issue?