Definition
Text classification assigns a text (a sentence, review, headline or document) one label from a fixed set , or a subset of it in the multi-label case. Deep-learning versions encode into a vector and put a linear layer on top: fastText averages embeddings (Joulin et al., 2017), Kim's CNN max-pools convolutional features (Kim, 2014), and the standard today is fine-tuning a pretrained encoder such as BERT (Devlin et al., 2018) on the [CLS] vector.
Topic labelling, spam filtering, intent detection, toxicity flags and sentiment are all the same task with a different label set. The input is variable-length text; the output is a distribution over classes chosen in advance.
What makes it hard: labels depend on a few words in context ("not bad", "chip" in a business story), classes overlap, and real label distributions are skewed. Before deep learning the standard pipeline was a bag of words or n-grams with TF-IDF weights feeding naive Bayes, logistic regression or a linear SVM. These baselines are still strong for topic labels.
Encoder plus a linear head
Every neural classifier has the same last step. Some encoder turns the token sequence into one vector , and a linear layer turns into logits. For a single label the logits go through a softmax, and training minimises cross-entropy:
- text representation: averaged embeddings, pooled CNN features, or the final [CLS] state for BERT-base
- row of the head and its bias
- the gold label
The encoders differ in what can capture. A fastText average ignores word order except for the local order kept by bigram features. Kim's CNN computes for each window of words and keeps , so it detects key phrases anywhere in the sentence. A fine-tuned Transformer contextualises every token, so "not" can change what "bad" contributes.
Fine-tuning BERT adds only the head ( parameters for BERT-base) and updates all weights end to end. The paper's recipe was 2–4 epochs, batch size 16 or 32, and learning rate 5e-5, 3e-5 or 2e-5, chosen on the dev set.
Multi-label: sigmoids, not softmax
If a text can carry several labels (a story about a chip maker's shares is Business and Sci/Tech), the softmax is the wrong output. It forces the probabilities to sum to 1, so adding a second label can only take mass away from the first. Instead each class gets its own binary decision:
- whether label is in the gold set
- decision threshold, 0.5 by default; often tuned per class on the dev set
Metrics
Accuracy is the fraction of texts with the right label. On skewed data it rewards predicting the majority class, so papers also report macro-F1, the unweighted mean of per-class F1. Micro-F1 pools the counts over classes first, so frequent classes dominate it.
- true positives, false positives and false negatives for class
Standard benchmarks
- AG News
- 4 news topics (World, Sports, Business, Sci/Tech); 120,000 train and 7,600 test examples, balanced (Zhang, Zhao and LeCun, 2015).
- SST-2
- Binary sentence sentiment from the Stanford Sentiment Treebank; GLUE split of 67,349 train, 872 dev and 1,821 test. BERT-base scores 93.5% and BERT-large 94.9% on the GLUE test set.
- IMDb
- 50,000 movie reviews, 25,000 train and 25,000 test, binary (Maas et al., 2011). Reviews are long enough that truncation matters; see Document classification.
- GLUE
- Nine sentence and sentence-pair tasks (Wang et al., 2018); most are classification, scored by accuracy, F1, or Matthews correlation for CoLA. STS-B is a regression task.
- Baseline
- fastText with bigrams reaches 92.5% on AG News and trains orders of magnitude faster than neural baselines, a useful floor before reaching for a Transformer.
Worked example: one headline, both heads
Take "chip maker shares jump on ai demand" from the figure. Every class starts at bias . The Business words add and the Sci/Tech words ("chip", "ai") add , so the logits are for (World, Sports, Business, Sci/Tech).
| Quantity | Computation | Value |
|---|---|---|
| softmax numerators | (0.0015, 0.0015, 1, 0.6065), sum 1.6095 | |
| softmax | divide by 1.6095 | (0.0009, 0.0009, 0.6213, 0.3768) |
| cross-entropy | 0.4759 | |
| sigmoid | (0.1192, 0.1192, 0.9890, 0.9820) | |
| BCE terms | twice, , | (0.1269, 0.1269, 0.0110, 0.0181) |
| BCE | sum, or mean over 4 labels (PyTorch default) | 0.2831, or 0.0708 |
| prediction at | classes with | {Business, Sci/Tech} |
The softmax splits the mass 62/38, so it predicts Business only and its 0.377 for Sci/Tech looks like doubt. The sigmoids say yes to both at 0.99 and 0.98: two near-certain labels, which is what the logits encode.
Macro-F1 at the default setting comes from the confusion matrix. World has TP 7, FN 1, so ; Business has TP 3, FP 1, so ; Sports and Sci/Tech are perfect. The mean is .
1import numpy as np2from datasets import load_dataset3from sklearn.metrics import accuracy_score, f1_score4from transformers import (AutoModelForSequenceClassification, AutoTokenizer,5 Trainer, TrainingArguments)67ds = load_dataset("ag_news") # 120,000 train / 7,600 test, 4 labels8tok = AutoTokenizer.from_pretrained("bert-base-uncased")9ds = ds.map(lambda b: tok(b["text"], truncation=True, max_length=128), batched=True)1011# adds a randomly initialised 4 x 768 head on the pooled [CLS] output12model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=4)13# multi-label instead: num_labels=K, problem_type="multi_label_classification"14# (BCE-with-logits on float 0/1 targets; threshold sigmoid(logits) at prediction time)1516def metrics(p):17 pred = np.argmax(p.predictions, axis=-1)18 return {"acc": accuracy_score(p.label_ids, pred),19 "macro_f1": f1_score(p.label_ids, pred, average="macro")}2021args = TrainingArguments("agnews-bert", learning_rate=2e-5, num_train_epochs=3,22 per_device_train_batch_size=32, eval_strategy="epoch")23Trainer(model=model, args=args, train_dataset=ds["train"], eval_dataset=ds["test"],24 processing_class=tok, compute_metrics=metrics).train()Use when
- +Linear and fastText models train in seconds and give a strong baseline.
- +Fine-tuned encoders bring pretrained knowledge of word order and context, which bag-of-words models lack.
- +The softmax/sigmoid head is a few thousand parameters; swapping label sets is cheap.
Avoid when
- −BERT-style encoders read at most 512 tokens; longer inputs need truncation or chunking.
- −Fixed label sets: a new class means new labels and retraining (or a zero-shot LLM prompt instead).
- −Accuracy hides failures on rare classes; report macro-F1 on skewed data.
Check yourself
A binary test set is 90% class A. A model always predicts A. What are its accuracy and macro-F1?
Check yourself
News articles may belong to any subset of 20 topics. Which output layer and loss fit?
Remember
- ✓Encoder → → linear head; for one label use with cross-entropy.
- ✓Encoders: fastText (averaged word + n-gram vectors), Kim's CNN (widths 3/4/5, max-over-time), fine-tuned BERT (the [CLS] vector).
- ✓Multi-label: per-class , binary cross-entropy, threshold tuned on dev data.
- ✓On skewed data, report macro-F1 alongside accuracy; a majority-class model can score high accuracy and near-zero macro-F1.
- ✓Worked example: logits (−2, −2, 4.5, 4) give softmax 0.62/0.38 but sigmoids 0.99/0.98.