Content generation

Production content generation runs brief → n drafts → automated checks → scorer → human editor → A/B test: quality comes from structured facts and a scorer that tracks real quality (gain ρ·E[max of n]), not from more drafts.

8 min read832 wordscore0 viewsupdated 2026-10-21

Definition

Content generation is the use of generative models to draft text meant for publication: product descriptions, marketing and ad copy, emails, social posts, landing pages and articles. A production system turns a structured brief (facts, audience, tone, length, forbidden claims) into several drafts, filters and ranks them automatically, and hands the best to a human editor who takes responsibility for what is published. There is no single founding paper; the parts come from Prompt templates, decoding control, LLM-as-a-judge and classic readability and A/B-testing practice.

The use case splits in two. Catalogue-scale generation writes thousands of near-identical items (one description per product, one email per segment) from database fields; the risk is a wrong spec repeated at scale. Editorial generation writes fewer, longer pieces (blog posts, reports); the risk is generic, unoriginal text and invented facts or quotes. Both need the same skeleton. The decoding side (diversity, repetition) is in Text generation, Temperature sampling and Repetition and frequency penalties; condensing a source document is Document summarization.

The pipeline

brief (structured input){ "product": "TrailLite 2 tent", "audience": "weekend hikers", "tone": "plain, friendly", "facts": ["1.2 kg", "3-minute pitch"], "max_words": 60, "never_say": ["lightest ever"]}LLM writes n = 5 drafts · automated checksfactsclaimswordspassd1Pitch it in 3 minutes. 1.2 kg.✓✓41passd2The lightest tent ever made.✓✗38dropd3Only 1.4 kg on your back.✗✓45dropd4Weekend trails, 1.2 kg pack.✓✓57passd5Adventure awaits! Our tent...✓✓83dropd3 contradicts the brief (1.4 ≠ 1.2 kg); d2 uses a banned claimscorerrank d1, d4(rubric judge)human editorfix, approve,own the claimspublish A/BA: d1 · B: d45,000 views eachmeasureCTR 4.0% vs 4.8%z = 1.95, p = 0.051results update the brief: winning angle, rejected claims, edits the editor keeps makinga 20% relative lift on 5,000 views per arm is not yet significant at the 5% level
fig.A JSON brief for a fictional tent drives five drafts. Cheap automated checks drop d2 (banned claim "lightest ever"), d3 (1.4 kg contradicts the 1.2 kg fact) and d5 (83 words > 60). A scorer ranks the survivors, an editor approves them, and an A/B test on 5,000 views per arm gives CTR 4.0% vs 4.8% (z = 1.95, p = 0.051). The results feed the next brief.
  1. 1Brief as data. Pass facts as fields (weight, price, dimensions, approved claims), not as prose for the model to remember. Every number in the output must trace back to a field. Structured output (JSON mode) helps when the output has slots (title, bullets, meta description).
  2. 2Generate n drafts at moderate temperature, varying the angle per draft (benefit, use case, objection) rather than relying on sampling noise alone.
  3. 3Check automatically: numbers and named entities against the brief, a banned-phrase list from legal and brand teams, length, reading level, and n-gram overlap with existing copy and with competitors' text.
  4. 4Score and rank the survivors with a rubric judge or a model trained on past editor decisions.
  5. 5Human edit and approve. The editor fixes, rejects or rewrites, and owns the claims. Log the edits: they are the best data for the next prompt or fine-tune.
  6. 6Publish and measure. Run variants as an A/B test on the metric that matters (click-through, conversion, unsubscribes), with a sample size fixed in advance.

Three numbers worth computing

FRE=206.835−1.015 wordssentences−84.6 syllableswords,FKGL=0.39 wordssentences+11.8 syllableswords−15.59\text{FRE}=206.835-1.015\,\frac{\text{words}}{\text{sentences}}-84.6\,\frac{\text{syllables}}{\text{words}},\qquad \text{FKGL}=0.39\,\frac{\text{words}}{\text{sentences}}+11.8\,\frac{\text{syllables}}{\text{words}}-15.59
FRE\text{FRE}
Flesch Reading Ease (Flesch, 1948): higher is easier; 60–70 is plain English, below 30 is very difficult
FKGL\text{FKGL}
Flesch–Kincaid Grade Level (Kincaid et al., 1975, for the US Navy): an approximate US school grade

Readability formulas only count sentence and word length. They catch the inflated register that LLMs drift into ("innovative", "seamless", "elevate") and are cheap enough to run on every draft. They say nothing about accuracy or persuasion, so use them as a gate, not a target.

si=ρ qi+1−ρ2 ei,E[qarg⁡max⁡isi]=ρ  E[max⁡i≤nZi]s_i=\rho\,q_i+\sqrt{1-\rho^2}\,e_i,\qquad \mathbb E\big[q_{\arg\max_i s_i}\big]=\rho\;\mathbb E\Big[\max_{i\le n}Z_i\Big]
qiq_i
true quality of draft i (what editors and readers would judge), standardised to N(0, 1)
sis_i
the automatic scorer's score; ρ is its correlation with true quality, e_i independent noise
E[max⁡Zi]\mathbb E[\max Z_i]
expected maximum of n standard normals: 0, 0.564, 1.029, 1.424, 1.766 for n = 1, 2, 4, 8, 16

This is the core trade-off of best-of-n drafting under a bivariate-normal model. Extra drafts raise the expected maximum only like 2ln⁡n\sqrt{2\ln n}, while a better scorer scales the whole gain linearly. When the scorer is itself a model that can be gamed (preferring length, keywords or flattery), pushing nn up selects for the scorer's quirks; Gao, Schulman and Hilton (2022) measured this for best-of-n against reward models, where gold-standard reward first rises and then falls as optimisation pressure grows.

z=p^B−p^Apˉ(1−pˉ)(1nA+1nB),pˉ=xA+xBnA+nBz=\frac{\hat p_B-\hat p_A}{\sqrt{\bar p(1-\bar p)\left(\frac1{n_A}+\frac1{n_B}\right)}},\qquad \bar p=\frac{x_A+x_B}{n_A+n_B}
p^A,p^B\hat p_A,\hat p_B
observed click-through or conversion rates of the two variants
x,  nx,\;n
conversions and impressions per arm; |z| > 1.96 is significant at the two-sided 5% level

Reference points

Flesch scale
90–100 very easy (about US grade 5); 60–70 plain English (grades 8–9); 0–30 very difficult (college graduate)
Diversity metrics
distinct-n, the share of unique n-grams across outputs (Li et al., NAACL 2016); Self-BLEU, the BLEU of each output against the others (Zhu et al., Texygen, SIGIR 2018): lower Self-BLEU means more varied drafts
Search guidance
Google Search Central (February 2023): content is judged on helpfulness, however it is produced, but generating content mainly to manipulate rankings breaks its spam policies; a March 2024 update named "scaled content abuse" explicitly
Fake reviews
US FTC final rule announced in August 2024 bans fake consumer reviews and testimonials, including ones generated by AI
Disclosure
EU AI Act (Regulation 2024/1689), Article 50: AI-generated text published to inform the public on matters of public interest must be disclosed unless it has undergone human review and someone holds editorial responsibility; these transparency duties apply from August 2026
one brief, n = 8 draftsshaded: editor accepts (q > 0.84)-2-20022scorer score s →true quality qP(editor accepts the pick)0.00.51.018162432drafts per brief nρ = 0.50: 0.443ρ = 1: 0.832random pick 0.200picked draft: s = 1.14, q = 1.09 → accepted (best draft in batch: q = 1.09)E[q of pick] = ρ·E[max of n normals] = 0.50 × 1.424 = 0.712 (perfect scorer: 1.424)P(accept) = 0.443 (one draft: 0.200 · perfect scorer at this n: 0.832)cost per piece = 8×$0.02 + $3 + (1 − 0.443)×$20 = $14.30cheapest n ≤ 32 at ρ = 0.50: n = 32 → $11.95 (n = 1: $19.02)q ~ N(0,1); scorer s = ρq + √(1−ρ²)e; editor accepts q > 0.84 (top 20%); costs are illustrative
fig.Drafts have true quality q ~ N(0, 1); the editor accepts q > 0.84 (the top 20% of single drafts). The pick is the draft with the highest scorer score. Defaults (n = 8, ρ = 0.5): expected quality of the pick 0.5 × 1.424 = 0.712, acceptance 0.443 (vs 0.200 for one draft, 0.832 with a perfect scorer), an illustrative 14.30 dollars per piece. At ρ = 0.9 the same 8 drafts give 0.730 and 8.56 dollars; at ρ = 0.5, quadrupling to n = 32 only reaches 0.584 (11.95 dollars). At ρ = 0 every n gives 0.200.

The cost model is 0.02n+3+20(1−Paccept)0.02n + 3 + 20(1-P_\text{accept}) dollars: drafts are almost free, a review costs 3 and a rejected pick costs a 20-dollar rewrite. Under these numbers more drafts never hurts, but going from a weak to a good scorer at n=8n=8 saves more than quadrupling nn. Spend effort on the scorer first: a rubric aligned with what editors reject, calibrated on their past decisions.

Worked example: readability and an A/B test

VersionWords / sentencesSyllablesFREFKGL
"This tent weighs just over a kilo. You can pitch it in three minutes with one pole."17 / 220206.835 − 8.628 − 99.529 = 98.71.6
"This innovative shelter delivers exceptional portability and can be deployed in approximately three minutes."14 / 133206.835 − 14.210 − 199.414 = −6.817.7
Two versions of the same copy, syllables counted by hand. Checked in Python.
  1. 1Variant A: 200 clicks in 5,000 views, so p̂_A = 0.040. Variant B: 240 in 5,000, p̂_B = 0.048: a 20% relative lift.
  2. 2Pooled rate p̄ = 440 / 10,000 = 0.044; standard error √(0.044 × 0.956 × 2/5,000) = 0.00410.
  3. 3z = 0.008 / 0.00410 = 1.95, two-sided p = 0.051: not significant at 5%, despite the large lift.
  4. 4To detect 4.0% → 4.8% with 80% power at α = 0.05 you need n ≈ (1.960 + 0.842)² × (0.0384 + 0.0457) / 0.008² ≈ 10,314 views per arm. Fix that before the test starts; stopping when p first dips below 0.05 inflates false positives.

In code

python
1import json, re
2
3product = {"name": "TrailLite 2 tent", "weight_kg": 1.2, "pitch_minutes": 3,
4 "audience": "weekend hikers"}
5NEVER_SAY = ["lightest ever", "waterproof", "guaranteed"]
6MAX_WORDS = 60
7
8def brief_prompt(p):
9 return [
10 {"role": "system", "content": (
11 "You write product copy. Use only the facts in the JSON. Plain, friendly tone. "
12 f"At most {MAX_WORDS} words. Never use: {', '.join(NEVER_SAY)}.")},
13 {"role": "user", "content": json.dumps(p)},
14 ]
15
16def check(draft, p):
17 """Cheap automated checks before any human sees the draft."""
18 issues = []
19 allowed = {str(v) for v in p.values() if isinstance(v, (int, float))}
20 for num in re.findall(r"\d+(?:\.\d+)?", draft):
21 if num not in allowed and num != "2": # "2" is part of the name
22 issues.append(f"number {num} not in the facts")
23 issues += [f"banned: {b}" for b in NEVER_SAY if b in draft.lower()]
24 if len(draft.split()) > MAX_WORDS:
25 issues.append(f"{len(draft.split())} words > {MAX_WORDS}")
26 return issues
27
28drafts = [ # stand-ins for n = 3 sampled completions of brief_prompt(product)
29 "The TrailLite 2 weighs 1.2 kg and pitches in 3 minutes, so you get to camp sooner.",
30 "The lightest ever tent for weekend hikers: 1.2 kg on your back.",
31 "At just 1.4 kg, the TrailLite 2 pitches in 3 minutes.",
32]
33for d in drafts:
34 print(check(d, product) or "pass", "|", d[:40])
35# pass | The TrailLite 2 weighs 1.2 kg and pitche
36# ['banned: lightest ever'] | The lightest ever tent for weekend hiker
37# ['number 1.4 not in the facts'] | At just 1.4 kg, the TrailLite 2 pitches
A brief rendered into chat messages, plus cheap pre-review checks (runs as is; the LLM call is omitted and three drafts are hard-coded). Real systems add entity checks, plagiarism overlap and a judge score.

Use when

  • +Drafting cost per item drops to cents, which makes per-product, per-segment and per-language copy affordable.
  • +Many variants per brief make A/B testing and personalisation practical.
  • +Structured briefs and checks make brand and legal rules executable instead of a style guide nobody reads.

Avoid when

  • −Output converges on a recognisable generic style; diversity needs explicit angles and measurement.
  • −Wrong facts scale with volume: one bad field can produce thousands of wrong pages.
  • −Editor time, not generation, becomes the bottleneck, and a weak scorer wastes it.

Check yourself

Your judge's scores correlate ρ = 0.5 with editors' ratings. Which change raises the expected quality of the chosen draft more: going from n = 8 to n = 32 drafts, or improving the judge to ρ = 0.9 at n = 8?

Check yourself

Variant B's click-through rate is 20% higher than A's (4.8% vs 4.0%) on 5,000 views each. What should you conclude?

Remember

  • ✓Pipeline: structured brief → n drafts → automated checks → scorer → human editor → A/B test → feedback into the brief.
  • ✓Every number and named entity in the output must come from the brief; invented specifics are the costliest failure.
  • ✓Best-of-n gain is ρ E[max⁡i≤nZi]\rho\,\mathbb E[\max_{i\le n}Z_i]: improve the scorer before raising nn, and watch for scorer gaming.
  • ✓Readability (Flesch, Flesch–Kincaid) and diversity (distinct-n, Self-BLEU) are cheap gates, not goals.
  • ✓Judge variants by a pre-registered A/B test: a 20% lift on 5,000 views per arm is still z = 1.95.
← back to Generative AI roadmap