Text classification

Map a text to labels from a fixed set: encode it into one vector (averaged embeddings, max-pooled CNN features or a fine-tuned [CLS] state), then a softmax head for one label or per-class sigmoids for many.

8 min read692 wordscore0 viewsupdated 2026-10-15

Definition

Text classification assigns a text xx (a sentence, review, headline or document) one label from a fixed set {1,…,K}\{1,\dots,K\}, or a subset of it in the multi-label case. Deep-learning versions encode xx into a vector hh and put a linear layer on top: fastText averages embeddings (Joulin et al., 2017), Kim's CNN max-pools convolutional features (Kim, 2014), and the standard today is fine-tuning a pretrained encoder such as BERT (Devlin et al., 2018) on the [CLS] vector.

Topic labelling, spam filtering, intent detection, toxicity flags and sentiment are all the same task with a different label set. The input is variable-length text; the output is a distribution over KK classes chosen in advance.

What makes it hard: labels depend on a few words in context ("not bad", "chip" in a business story), classes overlap, and real label distributions are skewed. Before deep learning the standard pipeline was a bag of words or n-grams with TF-IDF weights feeding naive Bayes, logistic regression or a linear SVM. These baselines are still strong for topic labels.

Encoder plus a linear head

Every neural classifier has the same last step. Some encoder turns the token sequence into one vector h∈Rdh\in\mathbb{R}^d, and a linear layer turns hh into KK logits. For a single label the logits go through a softmax, and training minimises cross-entropy:

p(y=k∣x)=exp⁡(wk⊤h+bk)∑j=1Kexp⁡(wj⊤h+bj),L=−log⁡p(y⋆∣x)p(y=k\mid x)=\frac{\exp(w_k^\top h+b_k)}{\sum_{j=1}^{K}\exp(w_j^\top h+b_j)},\qquad \mathcal{L}=-\log p(y^\star\mid x)
hh
text representation: averaged embeddings, pooled CNN features, or the final [CLS] state h[CLS]∈R768h_{\text{[CLS]}}\in\mathbb{R}^{768} for BERT-base
wk, bkw_k,\,b_k
row kk of the head W∈RK×dW\in\mathbb{R}^{K\times d} and its bias
y⋆y^\star
the gold label
“chip maker shares jump on ai demand”gold: BusinessfastText (Joulin et al. 2017)look up word + hashedbigram embeddingsaverage themh = mean of n vectorslinear layerz = W h + bsoftmax (hierarchicalfor many classes)CNN (Kim 2014)word2vec 300-d vectorssentence = n × 300 matrixconv filters, widths 3/4/5100 feature maps eachmax-over-time pooling→ 300-d sentence vectordropout 0.5 → linear→ softmaxfine-tuned encoder (BERT 2018)[CLS] chip maker sharesjump on ai demand [SEP]Transformer encoder12 layers, H = 768take h_[CLS] ∈ ℝ^768dropout 0.1W ∈ ℝ^(4×768) → softmaxall weights fine-tunedsingle-label: softmax, Σp = 1, argmaxmulti-label: σ per class, keep p ≥ 0.5World0.00World0.12Sports0.00Sports0.12Business0.62 ✓Business0.99 ✓Sci/Tech0.38Sci/Tech0.98 ✓predicts {Business} · loss: cross-entropypredicts {Business, Sci/Tech} · loss: BCE
fig.Three generations of encoder for the same 4-way AG News head. fastText averages word and hashed-bigram vectors; Kim's CNN slides 3-, 4- and 5-word filters over word2vec vectors and keeps each filter's maximum; BERT reads the whole sentence with self-attention and hands its [CLS] vector to the head. At the bottom, the same four logits read as a softmax (one label) or as independent sigmoids (any subset).

The encoders differ in what hh can capture. A fastText average ignores word order except for the local order kept by bigram features. Kim's CNN computes ci=f(w⋅xi:i+n−1+b)c_i=f(w\cdot x_{i:i+n-1}+b) for each window of nn words and keeps c^=max⁡ici\hat c=\max_i c_i, so it detects key phrases anywhere in the sentence. A fine-tuned Transformer contextualises every token, so "not" can change what "bad" contributes.

Fine-tuning BERT adds only the head (K×768+KK\times768+K parameters for BERT-base) and updates all weights end to end. The paper's recipe was 2–4 epochs, batch size 16 or 32, and learning rate 5e-5, 3e-5 or 2e-5, chosen on the dev set.

Multi-label: sigmoids, not softmax

If a text can carry several labels (a story about a chip maker's shares is Business and Sci/Tech), the softmax is the wrong output. It forces the probabilities to sum to 1, so adding a second label can only take mass away from the first. Instead each class gets its own binary decision:

pk=σ(wk⊤h+bk),L=−∑k=1K[yklog⁡pk+(1−yk)log⁡(1−pk)],Y^={k:pk≥τ}p_k=\sigma(w_k^\top h+b_k),\qquad \mathcal{L}=-\sum_{k=1}^{K}\Big[y_k\log p_k+(1-y_k)\log(1-p_k)\Big],\qquad \hat Y=\{k: p_k\ge\tau\}
yk∈{0,1}y_k\in\{0,1\}
whether label kk is in the gold set
τ\tau
decision threshold, 0.5 by default; often tuned per class on the dev set

Metrics

Accuracy is the fraction of texts with the right label. On skewed data it rewards predicting the majority class, so papers also report macro-F1, the unweighted mean of per-class F1. Micro-F1 pools the counts over classes first, so frequent classes dominate it.

F1k=2 TPk2 TPk+FPk+FNk,macro-F1=1K∑k=1KF1k\mathrm{F1}_k=\frac{2\,\mathrm{TP}_k}{2\,\mathrm{TP}_k+\mathrm{FP}_k+\mathrm{FN}_k},\qquad \text{macro-F1}=\frac1K\sum_{k=1}^{K}\mathrm{F1}_k
TPk,FPk,FNk\mathrm{TP}_k,\mathrm{FP}_k,\mathrm{FN}_k
true positives, false positives and false negatives for class kk

Standard benchmarks

AG News
4 news topics (World, Sports, Business, Sci/Tech); 120,000 train and 7,600 test examples, balanced (Zhang, Zhao and LeCun, 2015).
SST-2
Binary sentence sentiment from the Stanford Sentiment Treebank; GLUE split of 67,349 train, 872 dev and 1,821 test. BERT-base scores 93.5% and BERT-large 94.9% on the GLUE test set.
IMDb
50,000 movie reviews, 25,000 train and 25,000 test, binary (Maas et al., 2011). Reviews are long enough that truncation matters; see Document classification.
GLUE
Nine sentence and sentence-pair tasks (Wang et al., 2018); most are classification, scored by accuracy, F1, or Matthews correlation for CoLA. STS-B is a regression task.
Baseline
fastText with bigrams reaches 92.5% on AG News and trains orders of magnitude faster than neural baselines, a useful floor before reaching for a Transformer.
z_k = b_k + Σ_words w(word)_k · p = softmax(z), predict argmaxgold label (first square) · headline · predictionun talks on ceasefire resume in genevaW✓striker scores twice as united win derbyS✓shares fall after bank cuts forecastB✓new chip doubles battery life of phonesT✓chip maker shares jump on ai demandB✓olympic committee bans russia over dopingS✓election results spark protests in capitalW✓president meets nato leaders on ukraineW✓oil prices climb as opec cuts outputB✓space agency launches climate satelliteT✓government fines tech giant over data privacyT✓minister warns of tariffs on chinese steelW✓rebels seize airport as talks collapseW✓pope visits flood victims in brazilW✓parliament votes to leave eu treatyW✓trade deal lifts markets in tokyoB✗W World · S Sports · B Business · T Sci/Techselected: softmax over classesWorld0.001z=−2.0Sports0.001z=−2.0Business0.621z=+4.5Sci/Tech0.377z=+4.0confusion matrix (rows gold, cols predicted)WldWldSptSptBusBusSciSci7010020000300003F10.931.000.861.00accuracy 0.938 (15/16)macro-F1 0.9488 of 16 gold labels are World: always predicting World scores accuracy 0.500 but macro-F1 0.167softmax makes classes compete: probabilities sum to 1 and exactly one label is predictedclick a headline to inspect it · the weights are hand-set for illustration
output layer
fig.Sixteen headlines, a bag-of-words linear model with hand-set weights, and two ways to read its logits. Single-label (default): one World headline, "trade deal lifts markets in tokyo", is predicted Business, so accuracy is 0.938 (15/16) and macro-F1 0.948. Push the World offset to +8 and every headline becomes World: accuracy only falls to 0.500, because 8 of 16 gold labels are World, but macro-F1 collapses to 0.167. Switch to multi-label: at τ = 0.5 exact match is 0.938 and micro-F1 0.978; at τ = 0.2 four headlines pick up a spurious extra label and exact match drops to 0.750.

Worked example: one headline, both heads

Take "chip maker shares jump on ai demand" from the figure. Every class starts at bias −2-2. The Business words add 0.5+1+3+0.5+1.5=6.50.5+1+3+0.5+1.5=6.5 and the Sci/Tech words ("chip", "ai") add 3+3=63+3=6, so the logits are z=(−2, −2, 4.5, 4)z=(-2,\,-2,\,4.5,\,4) for (World, Sports, Business, Sci/Tech).

QuantityComputationValue
softmax numeratorsezk−4.5e^{z_k-4.5}(0.0015, 0.0015, 1, 0.6065), sum 1.6095
softmax ppdivide by 1.6095(0.0009, 0.0009, 0.6213, 0.3768)
cross-entropy−log⁡0.6213-\log 0.62130.4759
sigmoid pkp_k1/(1+e−zk)1/(1+e^{-z_k})(0.1192, 0.1192, 0.9890, 0.9820)
BCE terms−log⁡(1−0.1192)-\log(1-0.1192) twice, −log⁡0.9890-\log0.9890, −log⁡0.9820-\log0.9820(0.1269, 0.1269, 0.0110, 0.0181)
BCEsum, or mean over 4 labels (PyTorch default)0.2831, or 0.0708
prediction at τ=0.5\tau=0.5classes with pk≥0.5p_k\ge0.5{Business, Sci/Tech}
Softmax vs sigmoid on the same logits (gold: Business in the single-label set, {Business, Sci/Tech} in the multi-label set)

The softmax splits the mass 62/38, so it predicts Business only and its 0.377 for Sci/Tech looks like doubt. The sigmoids say yes to both at 0.99 and 0.98: two near-certain labels, which is what the logits encode.

Macro-F1 at the default setting comes from the confusion matrix. World has TP 7, FN 1, so F1=14/15=0.933\mathrm{F1}=14/15=0.933; Business has TP 3, FP 1, so F1=6/7=0.857\mathrm{F1}=6/7=0.857; Sports and Sci/Tech are perfect. The mean is (0.933+1+0.857+1)/4=0.948(0.933+1+0.857+1)/4=0.948.

python
1import numpy as np
2from datasets import load_dataset
3from sklearn.metrics import accuracy_score, f1_score
4from transformers import (AutoModelForSequenceClassification, AutoTokenizer,
5 Trainer, TrainingArguments)
6
7ds = load_dataset("ag_news") # 120,000 train / 7,600 test, 4 labels
8tok = AutoTokenizer.from_pretrained("bert-base-uncased")
9ds = ds.map(lambda b: tok(b["text"], truncation=True, max_length=128), batched=True)
10
11# adds a randomly initialised 4 x 768 head on the pooled [CLS] output
12model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=4)
13# multi-label instead: num_labels=K, problem_type="multi_label_classification"
14# (BCE-with-logits on float 0/1 targets; threshold sigmoid(logits) at prediction time)
15
16def metrics(p):
17 pred = np.argmax(p.predictions, axis=-1)
18 return {"acc": accuracy_score(p.label_ids, pred),
19 "macro_f1": f1_score(p.label_ids, pred, average="macro")}
20
21args = TrainingArguments("agnews-bert", learning_rate=2e-5, num_train_epochs=3,
22 per_device_train_batch_size=32, eval_strategy="epoch")
23Trainer(model=model, args=args, train_dataset=ds["train"], eval_dataset=ds["test"],
24 processing_class=tok, compute_metrics=metrics).train()
Fine-tuning BERT-base on AG News with Hugging Face, reporting accuracy and macro-F1 (not run here)

Use when

  • +Linear and fastText models train in seconds and give a strong baseline.
  • +Fine-tuned encoders bring pretrained knowledge of word order and context, which bag-of-words models lack.
  • +The softmax/sigmoid head is a few thousand parameters; swapping label sets is cheap.

Avoid when

  • −BERT-style encoders read at most 512 tokens; longer inputs need truncation or chunking.
  • −Fixed label sets: a new class means new labels and retraining (or a zero-shot LLM prompt instead).
  • −Accuracy hides failures on rare classes; report macro-F1 on skewed data.

Check yourself

A binary test set is 90% class A. A model always predicts A. What are its accuracy and macro-F1?

Check yourself

News articles may belong to any subset of 20 topics. Which output layer and loss fit?

Remember

  • ✓Encoder → hh → linear head; for one label use softmax(Wh+b)\mathrm{softmax}(Wh+b) with cross-entropy.
  • ✓Encoders: fastText (averaged word + n-gram vectors), Kim's CNN (widths 3/4/5, max-over-time), fine-tuned BERT (the [CLS] vector).
  • ✓Multi-label: per-class σ\sigma, binary cross-entropy, threshold τ\tau tuned on dev data.
  • ✓On skewed data, report macro-F1 alongside accuracy; a majority-class model can score high accuracy and near-zero macro-F1.
  • ✓Worked example: logits (−2, −2, 4.5, 4) give softmax 0.62/0.38 but sigmoids 0.99/0.98.
← back to Deep Learning roadmap