Keep Caer Bronach, "Cannot open from this side"

So i'm doing a quest to claim and clear a keep called Caer Bronach and when i enter the second inner courtyard i engage the enemy bandits, at my first time the bandits in the upper sanctum come down ...