I tried to replicate this and could not. My ladders placed just fine in a tower in the same configuration as yours with 2 tile wide walls on 4 sides and 1 tile wide walls on the others. It worked for 1 tile wide walls on all sides and 2 tile wide walls on all sides as well. I also tried with deleting the floor on the lower level and using a 2 story ladder and by leaving all the floors in and using 1 story ladders.
The only way I could not get it to work is when I tried to stack 2 ladders on top of each other. You have to offset the ladders in that case as you cannot stack them on top of each other.
Hope this helps.
