Definition
Named entity recognition (NER) is the task of finding the spans of a text that name real-world entities and assigning each span a type, such as person (PER), organisation (ORG), location (LOC) or miscellaneous (MISC) in the CoNLL-2003 shared task (Tjong Kim Sang and De Meulder, 2003). The term comes from the MUC-6 evaluation (1995). The standard neural recipe is a sequence labeller: tag every token with a BIO label, then read the spans off the tags.
Input: a tokenised sentence. Output: a set of (start, end, type) spans, e.g. {(Angela Merkel, PER), (Deutsche Bank, ORG), (New York, LOC)}. Spans have variable length, they cannot overlap in the flat setting, and the same string changes type with context: "Washington" is a person, a city or a government.
Before deep learning, NER systems were HMMs, maximum-entropy taggers and CRFs over hand-built features: capitalisation, prefixes and suffixes, word shape, part-of-speech tags and gazetteers (lists of known names). The best CoNLL-2003 English system (Florian et al., 2003) combined several such classifiers and reached 88.76 F1.
Spans as tags: BIO and BIOES
The span set is encoded as one label per token. In BIO (IOB2), B-X opens an entity of type , I-X continues it, and O is outside any entity. With 4 types that is labels. BIOES (also called BILOU) adds E-X for the last token and S-X for single-token entities, labels. It makes boundaries explicit and often helps slightly.
Not every tag sequence is valid. O I-PER is illegal in BIO, because I-PER must follow B-PER or I-PER. A tagger that predicts each token independently can emit such sequences; a CRF output layer learns to avoid them.
BiLSTM-CRF
Lample, Ballesteros, Subramanian, Kawakami and Dyer ("Neural Architectures for Named Entity Recognition", NAACL 2016) built each word vector from a pre-trained embedding plus a character-level BiLSTM, which handles capitalisation and unseen names. A word-level Bidirectional LSTM then produces emission scores for every token and tag . Huang, Xu and Yu (2015) proposed the same BiLSTM-CRF combination for tagging.
The CRF (Lafferty, McCallum and Pereira, 2001) scores the whole tag sequence with a learned transition matrix and normalises over all sequences:
- emission score of tag at token , from the BiLSTM (or BERT) output through a linear layer
- transition score from tag to ; and are added start and end tags
- log-partition over all tag sequences, computed by the forward algorithm in
Training maximises of the gold tags. Prediction is , found exactly by Viterbi decoding in (see Part-of-speech tagging). Because is learned to be very negative, illegal sequences lose. The model reached 90.94 F1 on CoNLL-2003 English without gazetteers; Ma and Hovy (2016) reported 91.21 with a character CNN instead of the character LSTM.
Fine-tuned transformers and subword alignment
BERT turns NER into token classification: a linear layer and softmax on each final hidden state, trained with cross-entropy, no CRF. Words are split into WordPieces, so labels must be aligned. BERT used the representation of each word's first sub-token. In the Hugging Face token-classification examples, the other pieces get label , which CrossEntropyLoss ignores.
- positions holding the first piece of a word (all others are labelled )
- final-layer hidden state of piece ; for BERT-base
- number of tags, 9 for CoNLL-2003 BIO
BERT reported CoNLL-2003 F1 of 92.4 (base) and 92.8 (large) on the test set, 96.4 and 96.6 on dev. A CRF layer on top of BERT is still a common option when tag-sequence validity matters, for example with BIOES or on small training sets.
Benchmarks and scoring
- CoNLL-2003 English
- Reuters news; 4 types (PER, ORG, LOC, MISC); 14,041 / 3,250 / 3,453 sentences in train / dev / test with 23,499 / 5,942 / 5,648 entities
- OntoNotes 5.0
- 18 entity types, including dates, money, quantities and cardinal numbers, across several genres
- Metric
- micro-averaged entity-level F1: a predicted span counts only if its start, end and type all match a gold span (the
conllevalscript;seqevalin Python) - Reference scores
- pre-neural 88.76 (Florian et al., 2003); BiLSTM-CRF 90.94; BiLSTM-CNN-CRF 91.21; BERT-large 92.8 (test F1)
Worked example: why the CRF fixes "I-PER I-PER"
Take "Angela Merkel", tags {O, B-PER, I-PER}. Emissions: Angela , Merkel . Per-token softmax gives Angela , so an independent tagger outputs the illegal I-PER I-PER. Add transitions: start→I-PER , O→I-PER , B→I , I→I , B→B and I→B , everything else 0.
| Tags | Score s(X, y) | p(y | X) |
|---|---|---|
| B-PER I-PER | 0 + 1.5 + 1.0 + 2.0 = 4.50 | 0.8441 |
| B-PER O | 0 + 1.5 + 0 + 0.2 = 1.70 | 0.0513 |
| B-PER B-PER | 0 + 1.5 − 1 + 0.8 = 1.30 | 0.0344 |
| O B-PER | 0 + 0.5 + 0 + 0.8 = 1.30 | 0.0344 |
| O O | 0 + 0.5 + 0 + 0.2 = 0.70 | 0.0189 |
| I-PER I-PER | −4 + 1.8 + 0.5 + 2.0 = 0.30 | 0.0127 |
| O I-PER, I-PER O, I-PER B-PER | −1.50, −2.00, −2.40 | 0.0021, 0.0013, 0.0009 |
Viterbi returns B-PER I-PER with probability 0.844. The emissions alone preferred I-PER at "Angela"; the start→I-PER penalty of overturns that, which is exactly what a CRF adds on top of per-token classification.
1def align_labels(word_labels, word_ids, label2id):2 """Label the first subword of each word; mask the rest and the special tokens with -100."""3 out, prev = [], None4 for w in word_ids: # from a fast tokenizer: enc.word_ids()5 out.append(-100 if w is None or w == prev else label2id[word_labels[w]])6 prev = w7 return out89def spans(tags):10 """BIO tags -> set of (start, end, type), conlleval-style (a stray I- opens a span)."""11 out, cur = set(), None12 for i, t in enumerate(tags + ["O"]):13 if cur and (t == "O" or t.startswith("B-") or t[2:] != cur[2]):14 out.add(tuple(cur)); cur = None15 if t != "O" and cur is None:16 cur = [i, i, t[2:]]17 elif cur:18 cur[1] = i19 return out2021def entity_f1(gold, pred):22 g, p = spans(gold), spans(pred)23 tp = len(g & p)24 prec, rec = tp / max(len(p), 1), tp / max(len(g), 1)25 return 2 * prec * rec / max(prec + rec, 1e-12)2627label2id = {"O": 0, "B-PER": 1, "I-PER": 2, "B-ORG": 3, "I-ORG": 4, "B-LOC": 5, "I-LOC": 6}28labels = ["B-PER", "I-PER", "O", "B-ORG", "I-ORG"] # Angela Merkel met Deutsche Bank29word_ids = [None, 0, 1, 1, 2, 3, 4, None] # [CLS] Angela Mer ##kel met Deutsche Bank [SEP]30print(align_labels(labels, word_ids, label2id)) # [-100, 1, 2, -100, 0, 3, 4, -100]3132gold = "B-PER I-PER O B-ORG I-ORG O O B-LOC I-LOC O O O B-ORG O O".split()33pred = "B-PER I-PER O O B-ORG O O B-LOC I-LOC O O O B-ORG O O".split() # "Deutsche" missed34acc = sum(a == b for a, b in zip(gold, pred)) / len(gold)35print(f"token acc {acc:.3f} entity F1 {entity_f1(gold, pred):.3f}") # token acc 0.867 entity F1 0.750Check yourself
Gold has "New York" as one LOC entity. The tagger outputs New = B-LOC, York = O. How is this scored at the entity level?
Check yourself
A softmax tagger outputs O, I-PER, I-PER for "met Angela Merkel". What would a CRF output layer change?
Remember
- ✓NER = tag each token with BIO (9 labels for CoNLL-2003) or BIOES (17), then decode spans.
- ✓BiLSTM-CRF: , trained with the forward algorithm, decoded with Viterbi; 90.94 F1 (Lample et al., 2016).
- ✓BERT: linear + softmax per token on each word's first subword, other pieces labelled −100; 92.8 test F1 for BERT-large.
- ✓Score with exact-match entity-level F1; token accuracy is inflated by the O class.
- ✓Flat BIO cannot represent nested entities; span-based or generative models can.