You may unwitting have another romance status unlocked with a different follower. Make the rounds through all the other open follower side quests you currently have unlocked, and chat them up to see if that's the case, and break off anything beyond friendship. If that doesnt work, try:
> Chat up his cross-dressing lieutentant, and do the drinking with the chargers thingee.
> If that doesnt work, you can try doing all the sit in judgement quests you have pending, to up his (and everyone else's) approval.
Hope that helps.
