Warden Grey's Final Repose - Can't access caves

This isn't accessible until much later in the game. It even says it in the quest description. As part of one of Taash's quests later you knock down a pole which lets you access that area, you can't be locked out of it unless the pole dissapears (which would be the bug ib question, but the journal descriptions suggests this is not the case)