how do you climb down a ladder in inquisition?

You should just get an interaction button for climbing up and down ladders, like with chests. If you send out a search ping and the ladder does not light up, then it is just background and cannot be used.

I have the interaction button, every time I "click" on it, it sends me back to the top & a few meteres away

EDIT: Nvm, didn't see your reply back

I just started the game, This is the very first ladder that you have to climb "down", i have climbed up ladders so far, np, but the 1st ladder down, party members are all at the top of the ladder, every time I try to interact with the ladder & go down, it puts me back at the top of the ladder a few meters away.

You can also just jump down, you don't need to climb down. You cannot die from jumping, you will just lose a lot of hit points (depending on the height of the jump, not much for a small drop) outside of your home base. In your home base, you can jump from any height without consequence.

Note if you land among a bunch of enemies  and start combat and lose enough hit points, then you may enter the knocked out state.

I  have this problem, too. It's not possible to jump from this location.

I'm in Easy mode - this worked fine in Normal, but there's been an update since I played the game last.

I find the ladders can be finicky.

(DAI, patch 7, PC)

Where you describe: try walking the character to near the top of the ladder, facing him away from it, then moving the camera angle and clicking the ladder when the marker turns gold.

The animation for going down ladder is delightful - the characters grab the edge and slide down to the bottom quickly (they don't climb down as such)