This is similar to an issue I've seen in my game for bookshelves on my vacation rental lots and libraries. Sims will grab a book to browse or read, when they get done, the book gets placed on a surface or ends up into their personal inventory rather than being returned to the bookcase. Another thing I am pretty sure happens is when you go to a library and you travel out, sims in the library who are holding books when you leave are missing the next time you go. If they don't go missing, then you find them on the floor where the sim was standing. Go there enough times and the bookshelf will eventually be empty. And since the book inventory is shared by all bookcases on the lot, you will only have the number of the default books that you have for one bookcase instead of the number you would have for multiple bookcases. To restock them, you have to go to build mode and buy one bookcase, drop it on the lot and delete one that was there so you have books again.
I don't know of any way you can stop NPCs from taking your books, but it may be cheaper to simply replace one bookcase which will replace all the default books all in one go rather than buying them individually. You might consider changing the business to a bookstore and then laying the books on sale on a surface rather than in a bookcase. NPCs would probably not pick them up to read but may only pick them up when they want to buy them. No idea if that will work as I have not tried either a library or bookstore as a small business though.
Hope this helps.