Definition
Coreference resolution is the task of finding every span of a document that refers to some entity (a mention) and partitioning the mentions into clusters, one per entity. The standard neural approach is the end-to-end span-ranking model of Lee, He, Lewis and Zettlemoyer (EMNLP 2017), which learns mention detection and antecedent linking jointly from raw text.
The task
Input: a document. Output: a set of clusters. In “Alice called Bob because she needed his car. He lent it to her.” the clusters are {Alice, she, her}, {Bob, his, He} and {his car, it}. Mentions can nest: “his” sits inside “his car” and belongs to a different entity.
It is hard because the cues are scattered. Gender and number agreement rule out some links, but choosing between two compatible candidates often needs world knowledge. In “The trophy doesn't fit in the suitcase because it is too big”, only physical knowledge says “it” is the trophy. This is the Winograd schema (Levesque et al. 2012).
Before deep learning came Hobbs' (1978) syntactic search for pronoun antecedents, then mention-pair classifiers over hand-built features (Soon, Ng and Lim 2001), then the deterministic multi-pass sieve of Stanford (Raghunathan et al. 2010). They all depended on a parser and a separate mention detector, so errors there could not be fixed later.
End-to-end span ranking
Lee et al. drop the pipeline. They consider every span up to words, give each one a vector, and let one network decide both whether it is a mention and which earlier span it points to. Each span picks an antecedent , where the dummy means “not a mention, or the first mention of its entity”.
- contextual state of token (a BiLSTM in 2017; BERT or SpanBERT later)
- attention-weighted sum of the span's word embeddings, a learned soft head word
- embedding of the span width
- mention score: how likely span is a mention at all
- antecedent score, ; holds speaker, genre and distance features
- fixed dummy score: a link is predicted only if some pair score is positive
There is no gold antecedent for each span, only gold clusters. So the loss marginalises over every correct choice:
- earlier spans in the gold cluster of ; if is not a mention or opens its cluster
Scoring all pairs of all spans would be in document length . Two prunes make it tractable: keep only the spans with the highest (), and let each consider at most antecedents. Clusters are the connected components of the argmax links.
Data, metrics and reference scores
- Benchmark
- CoNLL-2012 shared task on English OntoNotes 5.0 (Pradhan et al. 2012): 2,802 train, 343 dev, 348 test documents across news, broadcast, conversation, web and other genres. Singletons are not annotated.
- MUC
- Vilain et al. 1995. Link-based: counts the links needed to join the response's pieces of each key cluster.
- B³
- Bagga and Baldwin 1998. Mention-based: per-mention precision and recall of its cluster's overlap, averaged.
- CEAF$_{\phi_4}$
- Luo 2005. Entity-based: best one-to-one alignment of key and response clusters, similarity .
- CoNLL F1
- Mean of the three F1 scores, the headline number.
- Test CoNLL F1
- Lee et al. 2017: 67.2 (single), 68.8 (ensemble). Lee et al. 2018 (coarse-to-fine, ELMo): 73.0. BERT-large (Joshi et al. 2019): 76.9. SpanBERT-large (Joshi et al. 2020): 79.6.
Worked example: the antecedent of “it”
| candidate j | ||||
|---|---|---|---|---|
| — | — | 0 (fixed) | 0.256 | |
| Alice | 1.0 | −3.5 | −2.1 | 0.031 |
| Bob | 1.0 | −2.4 | −1.0 | 0.094 |
| she | 0.8 | −3.5 | −2.3 | 0.026 |
| his | 0.6 | −3.0 | −2.0 | 0.035 |
| his car | 0.7 | −0.4 | 0.7 | 0.516 |
| He | 0.8 | −3.0 | −1.8 | 0.042 |
The argmax is “his car”, the gold antecedent, so the loss for this span is . For “her” two gold antecedents exist, Alice (0.377) and she (0.508). The loss sums them, , so the model is not penalised for spreading mass across correct choices. Averaged over all eight mentions the loss is 0.299.
Scoring the over-merged output at Δ = +0.25
- 1Response: {Alice, she, her} and {Bob, his, his car, He, it}. Key: {Alice, she, her}, {Bob, his, He}, {his car, it}.
- 2MUC: no key link is missing, so recall is . The big response cluster spans 2 key clusters, so it has of its 4 links right. Precision is and F1 is 0.909.
- 3B³: recall is 1. Precision is , so F1 is 0.824.
- 4CEAF: the best alignment pairs {Alice, she, her} with itself () and {Bob, his, He} with the big cluster (). The total is 1.75, so recall is , precision is , and F1 is 0.700.
- 5CoNLL F1 .
1import torch2import torch.nn.functional as F34def antecedent_loss(s_m, s_a, gold):5 """s_m: (N,) mention scores; s_a: (N, N) antecedent scores, only j < i used;6 gold: (N, N+1) bool, column 0 = dummy antecedent. Returns loss and argmax links."""7 N = s_m.size(0)8 pair = s_m[:, None] + s_m[None, :] + s_a # s(i, j)9 earlier = torch.ones(N, N).tril(-1).bool()10 pair = pair.masked_fill(~earlier, float("-inf"))11 scores = torch.cat([torch.zeros(N, 1), pair], dim=1) # s(i, eps) = 012 logp = F.log_softmax(scores, dim=1)13 marg = torch.logsumexp(logp.masked_fill(~gold, float("-inf")), dim=1)14 return -marg.mean(), scores.argmax(dim=1) - 1 # -1 means eps1516# Alice Bob she his "his car" He it her17s_m = torch.tensor([1.0, 1.0, 0.8, 0.6, 0.7, 0.8, 0.4, 0.8])18lower = [[-3.2], [0.5, -2.5], [-2.5, 0.2, -2.8], [-3.0, -2.0, -3.0, -3.5],19 [-2.6, 0.4, -3.0, 0.6, -3.2], [-3.5, -2.4, -3.5, -3.0, -0.4, -3.0],20 [0.1, -2.8, 0.6, -2.9, -3.5, -3.1, -3.6]]21s_a = torch.zeros(8, 8)22for i, row in enumerate(lower, start=1):23 s_a[i, :i] = torch.tensor(row)2425gold = torch.zeros(8, 9, dtype=torch.bool)26for cluster in [[0, 2, 7], [1, 3, 5], [4, 6]]:27 for k, i in enumerate(cluster):28 gold[i, 0] = k == 0 # first mention: eps is correct29 for j in cluster[:k]:30 gold[i, j + 1] = True3132loss, ante = antecedent_loss(s_m, s_a, gold)33print(ante.tolist()) # [-1, -1, 0, 1, -1, 1, 4, 2]34print(round(loss.item(), 3)) # 0.299Use when
- +No parser or separate mention detector; one model, one loss
- +Swapping the encoder (BiLSTM → BERT → SpanBERT) lifted CoNLL F1 from 67.2 to 79.6 with the same head
- +Marginal loss needs only gold clusters, not one chosen antecedent per mention
Avoid when
- −Span enumeration is quadratic in length, so long documents need pruning or segmenting
- −Decisions are local (one antecedent per span); a cluster can become inconsistent through chains of links
- −Winograd-style cases still need world knowledge that local features lack
Check yourself
A span has pair scores −0.3, −1.2 and −2.0 to its three candidate antecedents. What does the model predict?
Check yourself
Why does the loss sum P over all gold antecedents instead of picking the nearest one?
Remember
- ✓Each span picks an antecedent from by softmax over , with .
- ✓Spans are ; pruning keeps spans () and up to antecedents.
- ✓Loss: −log of the total probability on correct antecedents, so only gold clusters are needed.
- ✓Evaluated on CoNLL-2012 (OntoNotes 5.0) by the mean F1 of MUC, B³ and CEAFφ4; 67.2 in 2017, 79.6 with SpanBERT.
- ✓Each metric has a bias (MUC favours merging), which is why the three are averaged.