Named entity recognition (NER)

NER finds and types entity spans by tagging every token in BIO; BiLSTM-CRF (Lample et al., 2016) reached 90.94 F1 on CoNLL-2003 and fine-tuned BERT-large 92.8, both scored by exact-match entity F1.

8 min read759 wordscore0 viewsupdated 2026-10-15

Definition

Named entity recognition (NER) is the task of finding the spans of a text that name real-world entities and assigning each span a type, such as person (PER), organisation (ORG), location (LOC) or miscellaneous (MISC) in the CoNLL-2003 shared task (Tjong Kim Sang and De Meulder, 2003). The term comes from the MUC-6 evaluation (1995). The standard neural recipe is a sequence labeller: tag every token with a BIO label, then read the spans off the tags.

Input: a tokenised sentence. Output: a set of (start, end, type) spans, e.g. {(Angela Merkel, PER), (Deutsche Bank, ORG), (New York, LOC)}. Spans have variable length, they cannot overlap in the flat setting, and the same string changes type with context: "Washington" is a person, a city or a government.

Before deep learning, NER systems were HMMs, maximum-entropy taggers and CRFs over hand-built features: capitalisation, prefixes and suffixes, word shape, part-of-speech tags and gazetteers (lists of known names). The best CoNLL-2003 English system (Florian et al., 2003) combined several such classifiers and reached 88.76 F1.

Spans as tags: BIO and BIOES

The span set is encoded as one label per token. In BIO (IOB2), B-X opens an entity of type XX, I-X continues it, and O is outside any entity. With 4 types that is 2⋅4+1=92\cdot4+1=9 labels. BIOES (also called BILOU) adds E-X for the last token and S-X for single-token entities, 4⋅4+1=174\cdot4+1=17 labels. It makes boundaries explicit and often helps slightly.

Not every tag sequence is valid. O I-PER is illegal in BIO, because I-PER must follow B-PER or I-PER. A tagger that predicts each token independently can emit such sequences; a CRF output layer learns to avoid them.

NER as token classification: tag every word, then read spans off the tagswords(input)AngelaMerkelmetDeutscheBankinNewYorkWordPiece(illustrative)[CLS]AngelaMer##kelmetDeutscheBankinNewYork[SEP]encodercontextual h_iBERT encoder (base: 12 layers, H = 768)2016 alternative: char-LSTM + word embedding → BiLSTM(Lample et al.); both give one vector h_i per tokentag head9 BIO labelssoftmax(W h_i + b), W ∈ ℝ^(9×768) · or a CRF: emissions + 9×9 transitionsloss = cross-entropy on first pieces only (−100 = ignored) · CRF: −log p(y | x)BIO tagsper piece-100B-PERI-PER-100OB-ORGI-ORGOB-LOCI-LOC-100entitiesB opens, I extendsPERORGLOCoutput: {(Angela Merkel, PER), (Deutsche Bank, ORG), (New York, LOC)}scored by exact-match entity F1 against the gold spans
fig.NER as token classification. The sentence is split into WordPieces; the encoder (BERT, or a BiLSTM over character and word embeddings in Lample et al.) gives one vector per piece; a 9-way softmax or a CRF predicts a BIO tag. Only each word's first piece carries a label: "##kel" and the special tokens get −100 and are ignored by the loss. B opens a span and I extends it.

BiLSTM-CRF

Lample, Ballesteros, Subramanian, Kawakami and Dyer ("Neural Architectures for Named Entity Recognition", NAACL 2016) built each word vector from a pre-trained embedding plus a character-level BiLSTM, which handles capitalisation and unseen names. A word-level Bidirectional LSTM then produces emission scores Pi,yP_{i,y} for every token ii and tag yy. Huang, Xu and Yu (2015) proposed the same BiLSTM-CRF combination for tagging.

The CRF (Lafferty, McCallum and Pereira, 2001) scores the whole tag sequence with a learned transition matrix AA and normalises over all KnK^n sequences:

s(X,y)=∑i=0nAyi, yi+1+∑i=1nPi, yi,log⁡p(y∣X)=s(X,y)−log⁡∑y~e s(X,y~)s(X,y)=\sum_{i=0}^{n}A_{y_i,\,y_{i+1}}+\sum_{i=1}^{n}P_{i,\,y_i},\qquad \log p(y\mid X)=s(X,y)-\log\sum_{\tilde y}e^{\,s(X,\tilde y)}
Pi,yP_{i,y}
emission score of tag yy at token ii, from the BiLSTM (or BERT) output through a linear layer
Ay,y′A_{y,y'}
transition score from tag yy to y′y'; y0y_0 and yn+1y_{n+1} are added start and end tags
log⁡∑y~\log\sum_{\tilde y}
log-partition over all tag sequences, computed by the forward algorithm in O(nK2)O(nK^2)

Training maximises log⁡p(y∣X)\log p(y\mid X) of the gold tags. Prediction is y∗=arg⁡max⁡ys(X,y)y^*=\arg\max_y s(X,y), found exactly by Viterbi decoding in O(nK2)O(nK^2) (see Part-of-speech tagging). Because AO, I-PERA_{\text{O},\,\text{I-PER}} is learned to be very negative, illegal sequences lose. The model reached 90.94 F1 on CoNLL-2003 English without gazetteers; Ma and Hovy (2016) reported 91.21 with a character CNN instead of the character LSTM.

Fine-tuned transformers and subword alignment

BERT turns NER into token classification: a linear layer and softmax on each final hidden state, trained with cross-entropy, no CRF. Words are split into WordPieces, so labels must be aligned. BERT used the representation of each word's first sub-token. In the Hugging Face token-classification examples, the other pieces get label −100-100, which CrossEntropyLoss ignores.

L=−1∣F∣∑i∈Flog⁡softmax⁡(Whi+b)yi,W∈RK×H\mathcal L=-\frac{1}{|\mathcal F|}\sum_{i\in\mathcal F}\log\operatorname{softmax}\big(Wh_i+b\big)_{y_i},\qquad W\in\mathbb R^{K\times H}
F\mathcal F
positions holding the first piece of a word (all others are labelled −100-100)
hih_i
final-layer hidden state of piece ii; H=768H=768 for BERT-base
KK
number of tags, 9 for CoNLL-2003 BIO

BERT reported CoNLL-2003 F1 of 92.4 (base) and 92.8 (large) on the test set, 96.4 and 96.6 on dev. A CRF layer on top of BERT is still a common option when tag-sequence validity matters, for example with BIOES or on small training sets.

Benchmarks and scoring

CoNLL-2003 English
Reuters news; 4 types (PER, ORG, LOC, MISC); 14,041 / 3,250 / 3,453 sentences in train / dev / test with 23,499 / 5,942 / 5,648 entities
OntoNotes 5.0
18 entity types, including dates, money, quantities and cardinal numbers, across several genres
Metric
micro-averaged entity-level F1: a predicted span counts only if its start, end and type all match a gold span (the conlleval script; seqeval in Python)
Reference scores
pre-neural 88.76 (Florian et al., 2003); BiLSTM-CRF 90.94; BiLSTM-CNN-CRF 91.21; BERT-large 92.8 (test F1)
prediction with 1 error · lenient (conlleval) decodinggoldwordpredB-PERAngelaB-PERI-PERMerkelI-PEROmetOB-ORGDeutscheOI-ORGBankB-ORGOexecutivesOOinOB-LOCNewB-LOCI-LOCYorkI-LOCgoldwordpredOonOOMondayOO,OB-ORGReutersB-ORGOreportedOO.Obars = decoded spans (PER ORG LOC MISC); dashed = no exact gold match · rose tag = token wrong · ! = illegal BIOtoken accuracy13/15 = 86.7%token F1 (non-O)0.769entity precision3/4 = 0.750entity recall3/4 = 0.750entity F10.750gold entities✓ PER Angela Merkel✗ ORG Deutsche Bank✓ LOC New York✓ ORG Reutersfalse positives+ ORG Banktoken accuracy 86.7% vs entity F1 75.0%: 8 of 15 tokens are O, so accuracy flatters the taggerentity-level = exact span and type (CoNLL-2003); a partly right span earns nothing
errors
decoding
fig.Toggle typical tagger errors and watch both metrics. At the default ("Deutsche" dropped from Deutsche Bank) token accuracy is 13/15 = 86.7% but entity F1 is 0.750, because the truncated span counts as one false positive and one miss. Tagging everything O still scores 53.3% accuracy with entity F1 0. Turn on only "I- without B": lenient conlleval decoding still recovers Angela Merkel (entity F1 1.000) while token F1 falls to 0.857; strict IOB2 decoding drops the span and F1 falls to 0.857.

Worked example: why the CRF fixes "I-PER I-PER"

Take "Angela Merkel", tags {O, B-PER, I-PER}. Emissions: Angela (0.5, 1.5, 1.8)(0.5,\,1.5,\,1.8), Merkel (0.2, 0.8, 2.0)(0.2,\,0.8,\,2.0). Per-token softmax gives Angela (0.135, 0.368, 0.497)(0.135,\,0.368,\,0.497), so an independent tagger outputs the illegal I-PER I-PER. Add transitions: start→I-PER =−4=-4, O→I-PER =−4=-4, B→I =1=1, I→I =0.5=0.5, B→B and I→B =−1=-1, everything else 0.

TagsScore s(X, y)p(y | X)
B-PER I-PER0 + 1.5 + 1.0 + 2.0 = 4.500.8441
B-PER O0 + 1.5 + 0 + 0.2 = 1.700.0513
B-PER B-PER0 + 1.5 − 1 + 0.8 = 1.300.0344
O B-PER0 + 0.5 + 0 + 0.8 = 1.300.0344
O O0 + 0.5 + 0 + 0.2 = 0.700.0189
I-PER I-PER−4 + 1.8 + 0.5 + 2.0 = 0.300.0127
O I-PER, I-PER O, I-PER B-PER−1.50, −2.00, −2.400.0021, 0.0013, 0.0009
All 9 sequences scored by s(X, y); log Z = 4.6695 (forward algorithm and brute force agree)

Viterbi returns B-PER I-PER with probability 0.844. The emissions alone preferred I-PER at "Angela"; the start→I-PER penalty of −4-4 overturns that, which is exactly what a CRF adds on top of per-token classification.

python
1def align_labels(word_labels, word_ids, label2id):
2 """Label the first subword of each word; mask the rest and the special tokens with -100."""
3 out, prev = [], None
4 for w in word_ids: # from a fast tokenizer: enc.word_ids()
5 out.append(-100 if w is None or w == prev else label2id[word_labels[w]])
6 prev = w
7 return out
8
9def spans(tags):
10 """BIO tags -> set of (start, end, type), conlleval-style (a stray I- opens a span)."""
11 out, cur = set(), None
12 for i, t in enumerate(tags + ["O"]):
13 if cur and (t == "O" or t.startswith("B-") or t[2:] != cur[2]):
14 out.add(tuple(cur)); cur = None
15 if t != "O" and cur is None:
16 cur = [i, i, t[2:]]
17 elif cur:
18 cur[1] = i
19 return out
20
21def entity_f1(gold, pred):
22 g, p = spans(gold), spans(pred)
23 tp = len(g & p)
24 prec, rec = tp / max(len(p), 1), tp / max(len(g), 1)
25 return 2 * prec * rec / max(prec + rec, 1e-12)
26
27label2id = {"O": 0, "B-PER": 1, "I-PER": 2, "B-ORG": 3, "I-ORG": 4, "B-LOC": 5, "I-LOC": 6}
28labels = ["B-PER", "I-PER", "O", "B-ORG", "I-ORG"] # Angela Merkel met Deutsche Bank
29word_ids = [None, 0, 1, 1, 2, 3, 4, None] # [CLS] Angela Mer ##kel met Deutsche Bank [SEP]
30print(align_labels(labels, word_ids, label2id)) # [-100, 1, 2, -100, 0, 3, 4, -100]
31
32gold = "B-PER I-PER O B-ORG I-ORG O O B-LOC I-LOC O O O B-ORG O O".split()
33pred = "B-PER I-PER O O B-ORG O O B-LOC I-LOC O O O B-ORG O O".split() # "Deutsche" missed
34acc = sum(a == b for a, b in zip(gold, pred)) / len(gold)
35print(f"token acc {acc:.3f} entity F1 {entity_f1(gold, pred):.3f}") # token acc 0.867 entity F1 0.750
Label alignment for subword models, and entity-level F1 versus token accuracy (pure Python, runs as is)

Check yourself

Gold has "New York" as one LOC entity. The tagger outputs New = B-LOC, York = O. How is this scored at the entity level?

Check yourself

A softmax tagger outputs O, I-PER, I-PER for "met Angela Merkel". What would a CRF output layer change?

Remember

  • ✓NER = tag each token with BIO (9 labels for CoNLL-2003) or BIOES (17), then decode spans.
  • ✓BiLSTM-CRF: s(X,y)=∑Ayi,yi+1+∑Pi,yis(X,y)=\sum A_{y_i,y_{i+1}}+\sum P_{i,y_i}, trained with the forward algorithm, decoded with Viterbi; 90.94 F1 (Lample et al., 2016).
  • ✓BERT: linear + softmax per token on each word's first subword, other pieces labelled −100; 92.8 test F1 for BERT-large.
  • ✓Score with exact-match entity-level F1; token accuracy is inflated by the O class.
  • ✓Flat BIO cannot represent nested entities; span-based or generative models can.
← back to Deep Learning roadmap