Nice approach! Reader engagement beats synthetic benchmarks any day. Bookmarked to try later - curious which models actually hook readers vs just score well on tests.

The plagiarism tumbler turns and turns.

Btw, creative writing is something where good sampler settings uniquely improve your experience a lot. That's why the coomer/ERP crowd is usually the first to implement a new sampler technique.

You should explore high temperature (far above 2) sampling with good truncations like min_p, top n sigma, TFS, mirostat, typicality sampling, etc. Basically anything that isn't top_p/top_k. This is the path to highly diverse outputs.

I run a site that does something similar, but on a more granular level (prompts at the page level rather than the chapter)

I think right now we're at the point where novelcrafter is an excellent proxy for the best models for readers, because LLMs are still mostly losing engagement due to technical errors as opposed to subjective ones:

That's repetition problems, moralizing/soft-censorship, grammatical quirks, missing instructions, forgetting major plot points, etc.

Those kinds of errors are so obvious you can almost rank these models with an N=1 vibe test, and they limit how much people will consume unless you're scratching certain itches like NSFW

-

However I do think with enough post-training you can beat that level of problems and move to a stage where the writing is technically sound (and that's what I've spent most of the last year working on).

From there you get to more challenging problems that require much more feedback along some level of specialization per user (like what Midjourney does during onboarding to build up a style profile). Once you're not making technical mistakes, you now have to codify the ethereal concept of "user taste", and that will be a really interesting challenge for LLMs.