If you hover over the book while it's in your sims inventory, it'll tell you what it is. "Best Seller" "Excellent", etc. It's best, when the option is giving to you, to submit Best Sellers to literary something or other. But, the quality of the book is determined when it's done being created, not on how it's published. You can also hover over the title when publishing and it'll tell you all the info on the book, including the quality.
I've had the best luck writing screenplays, but I have noticed it's whatever you have wrote the most of. So if your sim has wrote more Fiction, just have her keep writing fiction. Take a thoughtful shower then cloudgaze until the inspired moodlet pops and if you have those Sims 3 bonus lamps, put out two to three of those inspired ones and turn on the aura to get your sim as inspired as you can.
