After a weekend of trying, I realised it's not just me. It's a bug. And it's not not the only one. You'll run into many situations like this. Don't play much beyond the basics and you'll be okay. Play beyond that and you are bound to run into bugs which will make your game unplayable and suck all of the fun from it. After three days I'm done with it now.
