Keep Caer Bronach, "Cannot open from this side"
So i'm doing a quest to claim and clear a keep called Caer Bronach and when i enter the second inner courtyard i engage the enemy bandits, at my first time the bandits in the upper sanctum come down and open a door to fight my crew. I decided to quick load, after the quick load i stealth attack the guards in the courtyard with Cole but there are no sign of the guards coming down trough the door. Now im stuck at the courtyard with a door saying "Cannot open from this side" and theres a ladder that cant be reached and needs to be let down from the other side of the door that wont let me trough. Am i doing something wrong here ? I have tried every possible entry point to the castle and when i try the sea level entrance from the outside it says i have to claim the keep first.