The only time visiting sims (non-klepto) will "accidentally" walk off with any of your possessions is if the object in question had some use, meaning some reason to be in the visitors' hands. So that could be books, soccer/footballs, laptops and tablets, unnamed baby dragons, your sims' toddlers (yikes!). But gems are, for this purpose, considered objects with no interactive usage attached so the guests should be keeping their grubby hands off of them. :)