# Content generation

> Production content generation runs brief → n drafts → automated checks → scorer → human editor → A/B test: quality comes from structured facts and a scorer that tracks real quality (gain ρ·E[max of n]), not from more drafts.

Source: https://www.mlfoundry.in/learn/content-generation  
Section: Generative AI › Applications  
Level: core · 8 min read · updated 2026-10-21

**Content generation:** is the use of generative models to draft text meant for publication: product descriptions, marketing and ad copy, emails, social posts, landing pages and articles. A production system turns a structured **brief** (facts, audience, tone, length, forbidden claims) into several drafts, filters and ranks them automatically, and hands the best to a human editor who takes responsibility for what is published. There is no single founding paper; the parts come from **Prompt templates**, decoding control, **LLM-as-a-judge** and classic readability and A/B-testing practice.

The use case splits in two. **Catalogue-scale** generation writes thousands of near-identical items (one description per product, one email per segment) from database fields; the risk is a wrong spec repeated at scale. **Editorial** generation writes fewer, longer pieces (blog posts, reports); the risk is generic, unoriginal text and invented facts or quotes. Both need the same skeleton. The decoding side (diversity, repetition) is in **Text generation**, **Temperature sampling** and **Repetition and frequency penalties**; condensing a source document is **Document summarization**.

## The pipeline

> Figure: A JSON brief for a fictional tent drives five drafts. Cheap automated checks drop d2 (banned claim "lightest ever"), d3 (1.4 kg contradicts the 1.2 kg fact) and d5 (83 words > 60). A scorer ranks the survivors, an editor approves them, and an A/B test on 5,000 views per arm gives CTR 4.0% vs 4.8% (z = 1.95, p = 0.051). The results feed the next brief.

1. **Brief as data.** Pass facts as fields (weight, price, dimensions, approved claims), not as prose for the model to remember. Every number in the output must trace back to a field. **Structured output (JSON mode)** helps when the output has slots (title, bullets, meta description).
2. **Generate n drafts** at moderate temperature, varying the angle per draft (benefit, use case, objection) rather than relying on sampling noise alone.
3. **Check automatically**: numbers and named entities against the brief, a banned-phrase list from legal and brand teams, length, reading level, and n-gram overlap with existing copy and with competitors' text.
4. **Score and rank** the survivors with a rubric judge or a model trained on past editor decisions.
5. **Human edit and approve.** The editor fixes, rejects or rewrites, and owns the claims. Log the edits: they are the best data for the next prompt or fine-tune.
6. **Publish and measure.** Run variants as an A/B test on the metric that matters (click-through, conversion, unsubscribes), with a sample size fixed in advance.

## Three numbers worth computing

$$
\text{FRE}=206.835-1.015\,\frac{\text{words}}{\text{sentences}}-84.6\,\frac{\text{syllables}}{\text{words}},\qquad \text{FKGL}=0.39\,\frac{\text{words}}{\text{sentences}}+11.8\,\frac{\text{syllables}}{\text{words}}-15.59
$$

- $\text{FRE}$: Flesch Reading Ease (Flesch, 1948): higher is easier; 60–70 is plain English, below 30 is very difficult
- $\text{FKGL}$: Flesch–Kincaid Grade Level (Kincaid et al., 1975, for the US Navy): an approximate US school grade

Readability formulas only count sentence and word length. They catch the inflated register that LLMs drift into ("innovative", "seamless", "elevate") and are cheap enough to run on every draft. They say nothing about accuracy or persuasion, so use them as a gate, not a target.

$$
s_i=\rho\,q_i+\sqrt{1-\rho^2}\,e_i,\qquad \mathbb E\big[q_{\arg\max_i s_i}\big]=\rho\;\mathbb E\Big[\max_{i\le n}Z_i\Big]
$$

- $q_i$: true quality of draft i (what editors and readers would judge), standardised to N(0, 1)
- $s_i$: the automatic scorer's score; ρ is its correlation with true quality, e_i independent noise
- $\mathbb E[\max Z_i]$: expected maximum of n standard normals: 0, 0.564, 1.029, 1.424, 1.766 for n = 1, 2, 4, 8, 16

This is the core trade-off of best-of-n drafting under a bivariate-normal model. Extra drafts raise the expected maximum only like $\sqrt{2\ln n}$, while a better scorer scales the whole gain linearly. When the scorer is itself a model that can be gamed (preferring length, keywords or flattery), pushing $n$ up selects for the scorer's quirks; Gao, Schulman and Hilton (2022) measured this for best-of-n against reward models, where gold-standard reward first rises and then falls as optimisation pressure grows.

$$
z=\frac{\hat p_B-\hat p_A}{\sqrt{\bar p(1-\bar p)\left(\frac1{n_A}+\frac1{n_B}\right)}},\qquad \bar p=\frac{x_A+x_B}{n_A+n_B}
$$

- $\hat p_A,\hat p_B$: observed click-through or conversion rates of the two variants
- $x,\;n$: conversions and impressions per arm; |z| > 1.96 is significant at the two-sided 5% level

**Reference points**

- **Flesch scale:** 90–100 very easy (about US grade 5); 60–70 plain English (grades 8–9); 0–30 very difficult (college graduate)
- **Diversity metrics:** distinct-n, the share of unique n-grams across outputs (Li et al., NAACL 2016); Self-BLEU, the BLEU of each output against the others (Zhu et al., Texygen, SIGIR 2018): lower Self-BLEU means more varied drafts
- **Search guidance:** Google Search Central (February 2023): content is judged on helpfulness, however it is produced, but generating content mainly to manipulate rankings breaks its spam policies; a March 2024 update named "scaled content abuse" explicitly
- **Fake reviews:** US FTC final rule announced in August 2024 bans fake consumer reviews and testimonials, including ones generated by AI
- **Disclosure:** EU AI Act (Regulation 2024/1689), Article 50: AI-generated text published to inform the public on matters of public interest must be disclosed unless it has undergone human review and someone holds editorial responsibility; these transparency duties apply from August 2026

> Figure: Drafts have true quality q ~ N(0, 1); the editor accepts q > 0.84 (the top 20% of single drafts). The pick is the draft with the highest scorer score. Defaults (n = 8, ρ = 0.5): expected quality of the pick 0.5 × 1.424 = 0.712, acceptance 0.443 (vs 0.200 for one draft, 0.832 with a perfect scorer), an illustrative 14.30 dollars per piece. At ρ = 0.9 the same 8 drafts give 0.730 and 8.56 dollars; at ρ = 0.5, quadrupling to n = 32 only reaches 0.584 (11.95 dollars). At ρ = 0 every n gives 0.200.

The cost model is $0.02n + 3 + 20(1-P_\text{accept})$ dollars: drafts are almost free, a review costs 3 and a rejected pick costs a 20-dollar rewrite. Under these numbers more drafts never hurts, but going from a weak to a good scorer at $n=8$ saves more than quadrupling $n$. Spend effort on the scorer first: a rubric aligned with what editors reject, calibrated on their past decisions.

## Worked example: readability and an A/B test

Two versions of the same copy, syllables counted by hand. Checked in Python.

| Version | Words / sentences | Syllables | FRE | FKGL |
| --- | --- | --- | --- | --- |
| "This tent weighs just over a kilo. You can pitch it in three minutes with one pole." | 17 / 2 | 20 | 206.835 − 8.628 − 99.529 = 98.7 | 1.6 |
| "This innovative shelter delivers exceptional portability and can be deployed in approximately three minutes." | 14 / 1 | 33 | 206.835 − 14.210 − 199.414 = −6.8 | 17.7 |

1. Variant A: 200 clicks in 5,000 views, so p̂_A = 0.040. Variant B: 240 in 5,000, p̂_B = 0.048: a 20% relative lift.
2. Pooled rate p̄ = 440 / 10,000 = 0.044; standard error √(0.044 × 0.956 × 2/5,000) = 0.00410.
3. z = 0.008 / 0.00410 = 1.95, two-sided p = 0.051: not significant at 5%, despite the large lift.
4. To detect 4.0% → 4.8% with 80% power at α = 0.05 you need n ≈ (1.960 + 0.842)² × (0.0384 + 0.0457) / 0.008² ≈ 10,314 views per arm. Fix that before the test starts; stopping when p first dips below 0.05 inflates false positives.

## In code

```python
import json, re

product = {"name": "TrailLite 2 tent", "weight_kg": 1.2, "pitch_minutes": 3,
           "audience": "weekend hikers"}
NEVER_SAY = ["lightest ever", "waterproof", "guaranteed"]
MAX_WORDS = 60

def brief_prompt(p):
    return [
        {"role": "system", "content": (
            "You write product copy. Use only the facts in the JSON. Plain, friendly tone. "
            f"At most {MAX_WORDS} words. Never use: {', '.join(NEVER_SAY)}.")},
        {"role": "user", "content": json.dumps(p)},
    ]

def check(draft, p):
    """Cheap automated checks before any human sees the draft."""
    issues = []
    allowed = {str(v) for v in p.values() if isinstance(v, (int, float))}
    for num in re.findall(r"\d+(?:\.\d+)?", draft):
        if num not in allowed and num != "2":          # "2" is part of the name
            issues.append(f"number {num} not in the facts")
    issues += [f"banned: {b}" for b in NEVER_SAY if b in draft.lower()]
    if len(draft.split()) > MAX_WORDS:
        issues.append(f"{len(draft.split())} words > {MAX_WORDS}")
    return issues

drafts = [  # stand-ins for n = 3 sampled completions of brief_prompt(product)
    "The TrailLite 2 weighs 1.2 kg and pitches in 3 minutes, so you get to camp sooner.",
    "The lightest ever tent for weekend hikers: 1.2 kg on your back.",
    "At just 1.4 kg, the TrailLite 2 pitches in 3 minutes.",
]
for d in drafts:
    print(check(d, product) or "pass", "|", d[:40])
# pass | The TrailLite 2 weighs 1.2 kg and pitche
# ['banned: lightest ever'] | The lightest ever tent for weekend hiker
# ['number 1.4 not in the facts'] | At just 1.4 kg, the TrailLite 2 pitches
```

A brief rendered into chat messages, plus cheap pre-review checks (runs as is; the LLM call is omitted and three drafts are hard-coded). Real systems add entity checks, plagiarism overlap and a judge score.

> **Invented specifics are the expensive failure:** A model asked for persuasive copy will supply persuasive details: a battery life, a certification, a customer quote, a "studies show". In advertising these are claims that must be substantiated, and fabricated reviews or testimonials are banned outright in some jurisdictions. Allow numbers and named entities only from the brief, and treat any other as a hard failure. See **Hallucination**.

> **Disclosure and rights:** Whether you must label AI-generated content depends on the jurisdiction, the channel and how much human editing it had. Provenance metadata and watermarks are covered in **Watermarking generated content**, and who owns or may reuse generated text in **Copyright and licensing**. This article is not legal advice; check the rules that apply to you with counsel.

**Strengths**

- Drafting cost per item drops to cents, which makes per-product, per-segment and per-language copy affordable.
- Many variants per brief make A/B testing and personalisation practical.
- Structured briefs and checks make brand and legal rules executable instead of a style guide nobody reads.

**Limitations**

- Output converges on a recognisable generic style; diversity needs explicit angles and measurement.
- Wrong facts scale with volume: one bad field can produce thousands of wrong pages.
- Editor time, not generation, becomes the bottleneck, and a weak scorer wastes it.

**Question:** Your judge's scores correlate ρ = 0.5 with editors' ratings. Which change raises the expected quality of the chosen draft more: going from n = 8 to n = 32 drafts, or improving the judge to ρ = 0.9 at n = 8?

- A. n = 32, because more drafts always wins
- B. ρ = 0.9 at n = 8: 0.9 × 1.424 = 1.28 vs 0.5 × 2.070 = 1.03
- C. They are equal
- D. Neither changes expected quality

**Answer:** B. ρ = 0.9 at n = 8: 0.9 × 1.424 = 1.28 vs 0.5 × 2.070 = 1.03 — Expected quality of the pick is $\rho\,\mathbb E[\max_{i\le n}Z_i]$. The expected maximum grows only like $\sqrt{2\ln n}$ (1.424 at $n=8$, 2.070 at $n=32$), but $\rho$ multiplies everything. Acceptance rates in the viz say the same: 0.730 vs 0.584.

**Question:** Variant B's click-through rate is 20% higher than A's (4.8% vs 4.0%) on 5,000 views each. What should you conclude?

- A. B is better; ship it
- B. The difference is not significant at the 5% level (z = 1.95); keep the test running to the sample size planned in advance
- C. A is better
- D. A/B tests cannot compare generated copy

**Answer:** B. The difference is not significant at the 5% level (z = 1.95); keep the test running to the sample size planned in advance — Pooled p̄ = 0.044, standard error 0.0041, z = 0.008 / 0.0041 = 1.95 < 1.96. About 10,300 views per arm are needed for 80% power at this effect size; deciding on a peek inflates the false-positive rate.

**Key points**

- Pipeline: structured brief → n drafts → automated checks → scorer → human editor → A/B test → feedback into the brief.
- Every number and named entity in the output must come from the brief; invented specifics are the costliest failure.
- Best-of-n gain is $\rho\,\mathbb E[\max_{i\le n}Z_i]$: improve the scorer before raising $n$, and watch for scorer gaming.
- Readability (Flesch, Flesch–Kincaid) and diversity (distinct-n, Self-BLEU) are cheap gates, not goals.
- Judge variants by a pre-registered A/B test: a 20% lift on 5,000 views per arm is still z = 1.95.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
