# Text classification

> Map a text to labels from a fixed set: encode it into one vector (averaged embeddings, max-pooled CNN features or a fine-tuned [CLS] state), then a softmax head for one label or per-class sigmoids for many.

Source: https://www.mlfoundry.in/learn/text-classification  
Section: Deep Learning › Deep Learning for NLP  
Level: core · 8 min read · updated 2026-10-15

**Text classification:** assigns a text $x$ (a sentence, review, headline or document) one label from a fixed set $\{1,\dots,K\}$, or a subset of it in the multi-label case. Deep-learning versions encode $x$ into a vector $h$ and put a linear layer on top: fastText averages embeddings (Joulin et al., 2017), Kim's CNN max-pools convolutional features (Kim, 2014), and the standard today is fine-tuning a pretrained encoder such as **BERT** (Devlin et al., 2018) on the [CLS] vector.

Topic labelling, spam filtering, intent detection, toxicity flags and sentiment are all the same task with a different label set. The input is variable-length text; the output is a distribution over $K$ classes chosen in advance.

What makes it hard: labels depend on a few words in context ("not bad", "chip" in a business story), classes overlap, and real label distributions are skewed. Before deep learning the standard pipeline was a bag of words or n-grams with TF-IDF weights feeding naive Bayes, logistic regression or a linear SVM. These baselines are still strong for topic labels.

## Encoder plus a linear head

Every neural classifier has the same last step. Some encoder turns the token sequence into one vector $h\in\mathbb{R}^d$, and a linear layer turns $h$ into $K$ logits. For a single label the logits go through a softmax, and training minimises cross-entropy:

$$
p(y=k\mid x)=\frac{\exp(w_k^\top h+b_k)}{\sum_{j=1}^{K}\exp(w_j^\top h+b_j)},\qquad \mathcal{L}=-\log p(y^\star\mid x)
$$

- $h$: text representation: averaged embeddings, pooled CNN features, or the final [CLS] state $h_{\text{[CLS]}}\in\mathbb{R}^{768}$ for BERT-base
- $w_k,\,b_k$: row $k$ of the head $W\in\mathbb{R}^{K\times d}$ and its bias
- $y^\star$: the gold label

> Figure: Three generations of encoder for the same 4-way AG News head. fastText averages word and hashed-bigram vectors; Kim's CNN slides 3-, 4- and 5-word filters over word2vec vectors and keeps each filter's maximum; BERT reads the whole sentence with self-attention and hands its [CLS] vector to the head. At the bottom, the same four logits read as a softmax (one label) or as independent sigmoids (any subset).

The encoders differ in what $h$ can capture. A fastText average ignores word order except for the local order kept by bigram features. Kim's CNN computes $c_i=f(w\cdot x_{i:i+n-1}+b)$ for each window of $n$ words and keeps $\hat c=\max_i c_i$, so it detects key phrases anywhere in the sentence. A fine-tuned Transformer contextualises every token, so "not" can change what "bad" contributes.

Fine-tuning **BERT** adds only the head ($K\times768+K$ parameters for BERT-base) and updates all weights end to end. The paper's recipe was 2–4 epochs, batch size 16 or 32, and learning rate 5e-5, 3e-5 or 2e-5, chosen on the dev set.

## Multi-label: sigmoids, not softmax

If a text can carry several labels (a story about a chip maker's shares is Business and Sci/Tech), the softmax is the wrong output. It forces the probabilities to sum to 1, so adding a second label can only take mass away from the first. Instead each class gets its own binary decision:

$$
p_k=\sigma(w_k^\top h+b_k),\qquad \mathcal{L}=-\sum_{k=1}^{K}\Big[y_k\log p_k+(1-y_k)\log(1-p_k)\Big],\qquad \hat Y=\{k: p_k\ge\tau\}
$$

- $y_k\in\{0,1\}$: whether label $k$ is in the gold set
- $\tau$: decision threshold, 0.5 by default; often tuned per class on the dev set

## Metrics

Accuracy is the fraction of texts with the right label. On skewed data it rewards predicting the majority class, so papers also report **macro-F1**, the unweighted mean of per-class F1. Micro-F1 pools the counts over classes first, so frequent classes dominate it.

$$
\mathrm{F1}_k=\frac{2\,\mathrm{TP}_k}{2\,\mathrm{TP}_k+\mathrm{FP}_k+\mathrm{FN}_k},\qquad \text{macro-F1}=\frac1K\sum_{k=1}^{K}\mathrm{F1}_k
$$

- $\mathrm{TP}_k,\mathrm{FP}_k,\mathrm{FN}_k$: true positives, false positives and false negatives for class $k$

**Standard benchmarks**

- **AG News:** 4 news topics (World, Sports, Business, Sci/Tech); 120,000 train and 7,600 test examples, balanced (Zhang, Zhao and LeCun, 2015).
- **SST-2:** Binary sentence sentiment from the Stanford Sentiment Treebank; GLUE split of 67,349 train, 872 dev and 1,821 test. BERT-base scores 93.5% and BERT-large 94.9% on the GLUE test set.
- **IMDb:** 50,000 movie reviews, 25,000 train and 25,000 test, binary (Maas et al., 2011). Reviews are long enough that truncation matters; see **Document classification**.
- **GLUE:** Nine sentence and sentence-pair tasks (Wang et al., 2018); most are classification, scored by accuracy, F1, or Matthews correlation for CoLA. STS-B is a regression task.
- **Baseline:** fastText with bigrams reaches 92.5% on AG News and trains orders of magnitude faster than neural baselines, a useful floor before reaching for a Transformer.

> Figure: Sixteen headlines, a bag-of-words linear model with hand-set weights, and two ways to read its logits. Single-label (default): one World headline, "trade deal lifts markets in tokyo", is predicted Business, so accuracy is 0.938 (15/16) and macro-F1 0.948. Push the World offset to +8 and every headline becomes World: accuracy only falls to 0.500, because 8 of 16 gold labels are World, but macro-F1 collapses to 0.167. Switch to multi-label: at τ = 0.5 exact match is 0.938 and micro-F1 0.978; at τ = 0.2 four headlines pick up a spurious extra label and exact match drops to 0.750.

## Worked example: one headline, both heads

Take "chip maker shares jump on ai demand" from the figure. Every class starts at bias $-2$. The Business words add $0.5+1+3+0.5+1.5=6.5$ and the Sci/Tech words ("chip", "ai") add $3+3=6$, so the logits are $z=(-2,\,-2,\,4.5,\,4)$ for (World, Sports, Business, Sci/Tech).

Softmax vs sigmoid on the same logits (gold: Business in the single-label set, {Business, Sci/Tech} in the multi-label set)

| Quantity | Computation | Value |
| --- | --- | --- |
| softmax numerators | $e^{z_k-4.5}$ | (0.0015, 0.0015, 1, 0.6065), sum 1.6095 |
| softmax $p$ | divide by 1.6095 | (0.0009, 0.0009, 0.6213, 0.3768) |
| cross-entropy | $-\log 0.6213$ | 0.4759 |
| sigmoid $p_k$ | $1/(1+e^{-z_k})$ | (0.1192, 0.1192, 0.9890, 0.9820) |
| BCE terms | $-\log(1-0.1192)$ twice, $-\log0.9890$, $-\log0.9820$ | (0.1269, 0.1269, 0.0110, 0.0181) |
| BCE | sum, or mean over 4 labels (PyTorch default) | 0.2831, or 0.0708 |
| prediction at $\tau=0.5$ | classes with $p_k\ge0.5$ | {Business, Sci/Tech} |

The softmax splits the mass 62/38, so it predicts Business only and its 0.377 for Sci/Tech looks like doubt. The sigmoids say yes to both at 0.99 and 0.98: two near-certain labels, which is what the logits encode.

Macro-F1 at the default setting comes from the confusion matrix. World has TP 7, FN 1, so $\mathrm{F1}=14/15=0.933$; Business has TP 3, FP 1, so $\mathrm{F1}=6/7=0.857$; Sports and Sci/Tech are perfect. The mean is $(0.933+1+0.857+1)/4=0.948$.

```python
import numpy as np
from datasets import load_dataset
from sklearn.metrics import accuracy_score, f1_score
from transformers import (AutoModelForSequenceClassification, AutoTokenizer,
                          Trainer, TrainingArguments)

ds = load_dataset("ag_news")                      # 120,000 train / 7,600 test, 4 labels
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
ds = ds.map(lambda b: tok(b["text"], truncation=True, max_length=128), batched=True)

# adds a randomly initialised 4 x 768 head on the pooled [CLS] output
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=4)
# multi-label instead: num_labels=K, problem_type="multi_label_classification"
# (BCE-with-logits on float 0/1 targets; threshold sigmoid(logits) at prediction time)

def metrics(p):
    pred = np.argmax(p.predictions, axis=-1)
    return {"acc": accuracy_score(p.label_ids, pred),
            "macro_f1": f1_score(p.label_ids, pred, average="macro")}

args = TrainingArguments("agnews-bert", learning_rate=2e-5, num_train_epochs=3,
                         per_device_train_batch_size=32, eval_strategy="epoch")
Trainer(model=model, args=args, train_dataset=ds["train"], eval_dataset=ds["test"],
        processing_class=tok, compute_metrics=metrics).train()
```

Fine-tuning BERT-base on AG News with Hugging Face, reporting accuracy and macro-F1 (not run here)

> **The threshold is a hyperparameter:** A multi-label model trained with BCE is rarely calibrated so that 0.5 is the best cut for every class. Rare labels usually need a lower threshold. Tune $\tau$ per class on the dev set, not on the test set, and report which one you used.

> **Topic is easy, meaning is hard:** On topic datasets like AG News, a bag of n-grams already captures most of the signal, so linear models land within a few points of Transformers. The gap widens when the label depends on composition (negation, contrast, sarcasm) or on two texts at once, as in GLUE's entailment tasks. **Sentiment analysis** covers the compositional case.

**Strengths**

- Linear and fastText models train in seconds and give a strong baseline.
- Fine-tuned encoders bring pretrained knowledge of word order and context, which bag-of-words models lack.
- The softmax/sigmoid head is a few thousand parameters; swapping label sets is cheap.

**Limitations**

- BERT-style encoders read at most 512 tokens; longer inputs need truncation or chunking.
- Fixed label sets: a new class means new labels and retraining (or a zero-shot LLM prompt instead).
- Accuracy hides failures on rare classes; report macro-F1 on skewed data.

**Question:** A binary test set is 90% class A. A model always predicts A. What are its accuracy and macro-F1?

- A. 0.90 and 0.90
- B. 0.90 and about 0.47
- C. 0.50 and 0.50
- D. 0.90 and 0.00

**Answer:** B. 0.90 and about 0.47 — Accuracy is 0.90. For class A, TP = 0.9 and FP = 0.1 (per unit of data), so $\mathrm{F1}_A=1.8/1.9=0.947$. Class B is never predicted, so $\mathrm{F1}_B=0$. Macro-F1 is $(0.947+0)/2=0.474$.

**Question:** News articles may belong to any subset of 20 topics. Which output layer and loss fit?

- A. 20-way softmax with cross-entropy, predict the argmax
- B. 20 independent sigmoids with binary cross-entropy, threshold each
- C. One sigmoid with BCE
- D. 20-way softmax, predict every class above 1/20

**Answer:** B. 20 independent sigmoids with binary cross-entropy, threshold each — Labels are not mutually exclusive, so each needs its own Bernoulli output. A softmax ties the probabilities together, so a text with two true topics can score at most about 0.5 on each.

**Key points**

- Encoder → $h$ → linear head; for one label use $\mathrm{softmax}(Wh+b)$ with cross-entropy.
- Encoders: fastText (averaged word + n-gram vectors), Kim's CNN (widths 3/4/5, max-over-time), fine-tuned BERT (the [CLS] vector).
- Multi-label: per-class $\sigma$, binary cross-entropy, threshold $\tau$ tuned on dev data.
- On skewed data, report macro-F1 alongside accuracy; a majority-class model can score high accuracy and near-zero macro-F1.
- Worked example: logits (−2, −2, 4.5, 4) give softmax 0.62/0.38 but sigmoids 0.99/0.98.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
