Having to 'Set as front door' every time I leave the lot

I've been having this problem as well. I have a large lot with a barn and a house. Every time I load the lot, it resets the barn door as the Front Door instead of the house door. The house and barn are not connected and are across the lot from each other.   All I can think is it happens because the barn door is closer to the lot edge than the house door. I feel that once I set a door as a front door it should remain set no matter the position of the other doors on the lot.