speedracer1177 it's designed as it would be in real life. You'd still need to unlock a real door to let in an invited guest.
If you have City Living you can give a key to non household Sims but if the door is locked for everyone but the household they still can't get in. But if you take that lock setting off and rely on your No Trespassing settings to keep out unwanted guests, your key holders will just walk in without the need to invite them in. But they can start visiting a bit too frequently and making themselves a bit too comfortable, they'll even turn up to your vacation lots.
I usually have outdoor seating for my Sims to socialise while the house remains locked. That saves me unlocking the door and means they don't hover on the doorstep looking annoyed. Or I set internal door locks around the house and leave one reception area unlocked.