# Information extraction

> Information extraction turns text into typed records (entities, relation triples, events, slots); pipelines classify marked entity pairs, generative models such as REBEL write the triples as text, and both are scored by strict triple-level F1.

Source: https://www.mlfoundry.in/learn/information-extraction  
Section: Deep Learning › Deep Learning for NLP  
Level: core · 8 min read · updated 2026-10-15

**Information extraction:** (IE) is the task of turning unstructured text into structured records with a fixed schema: **entities** (see **Named entity recognition (NER)**), **relations** between them as (head, relation, tail) triples, **events** (a trigger word plus arguments in named roles), and **slots** of a template about a given entity. The task was shaped by the MUC evaluations (1987–1997) and the ACE program; ACE 2005 is still a standard benchmark.

Input: a sentence or document. Output: a set of records. From "Marie Curie was born in Warsaw" a system should produce (Marie Curie, born_in, Warsaw). The hard parts: the same relation is phrased in many ways, most entity pairs in a text are **not** related, arguments may be pronouns ("she"), and the relation may span sentences.

Before deep learning, relation extraction used hand-written patterns and feature-based or kernel classifiers over words, entity types and dependency paths. **Distant supervision** (Mintz et al., 2009) created training data by labelling every sentence that mentions a Freebase entity pair with that pair's relation, which is cheap but noisy.

> Figure: Four IE outputs from one sentence: entities, relation triples, an ACE-style Life:Marry event (trigger "married" with Person and Time arguments), and template slots for the query entity "Marie Curie". Below, the two ways to produce the triples: a pipeline that marks each entity pair and classifies it, or a sequence-to-sequence model that writes the triples as a linearized string to be parsed.

## The subtasks

| Subtask | Output | Typical neural model |
| --- | --- | --- |
| Relation extraction | (head, relation, tail) or no_relation per entity pair | encoder + classifier on the two entity representations |
| Event extraction | trigger span + type; arguments with roles (Person, Time, Place, …) | trigger tagging, then argument classification per (trigger, entity) |
| Slot filling | values for a fixed set of slots: of a query entity (KBP), or of an utterance (dialogue, e.g. ATIS) | relation classifier per candidate (KBP); BIO tagging (dialogue) |
| Open IE | (arg1, relation phrase, arg2) with no fixed schema | TextRunner (Banko et al., 2007), ReVerb (Fader et al., 2011); later neural taggers and seq2seq |

## Pipeline relation extraction with entity markers

The pipeline runs NER first, then forms every candidate pair and classifies it. The standard input inserts marker tokens around the two entities, e.g. `[E1] Marie Curie [/E1] was born in [E2] Warsaw [/E2]`. Baldini Soares et al. (2019, "Matching the Blanks") found that concatenating the encoder states at the two start markers works best:

$$
p(r\mid x,e_1,e_2)=\operatorname{softmax}\big(W\,[\,h_{\langle E1\rangle};\,h_{\langle E2\rangle}\,]+b\big)_r,\qquad r\in\mathcal R\cup\{\texttt{no\_relation}\}
$$

- $h_{\langle E1\rangle},h_{\langle E2\rangle}$: final encoder states (e.g. **BERT**) at the start markers of head and tail
- $\mathcal R$: the schema's relation types; `no_relation` absorbs the many unrelated pairs
- $W\in\mathbb R^{(|\mathcal R|+1)\times2H}$: a linear layer trained with cross-entropy

Each pair needs a forward pass, and NER errors **propagate**: a pair containing a missed or mis-bounded entity can never be correct. If each entity were found exactly with probability 0.9, independently, a pair would survive with $0.9^2=0.81$. **Joint** models predict entities and relations together, e.g. Miwa and Bansal (2016) with LSTMs over sequences and dependency trees. Zhong and Chen (2021, "A Frustratingly Easy Approach for Entity and Relation Extraction") then showed that a pipeline with separate encoders and **typed** markers matches or beats joint models on ACE and SciERC.

## Generative IE

A sequence-to-sequence model (see **BART**, **T5**) can write the records as text: train $p(y\mid x)$ where $y$ is a linearized list of triples, then parse $y$. REBEL (Huguet Cabot and Navigli, 2021) fine-tunes BART-large on the format `<triplet> head <subj> tail <obj> relation`. TANL (Paolini et al., 2021) writes augmented copies of the input, and UIE (Lu et al., 2022) uses one structured language for entities, relations, events and sentiment.

$$
p(y\mid x)=\prod_{t=1}^{|y|}p\big(y_t\mid y_{<t},x\big),\qquad P=\frac{|\hat T\cap T|}{|\hat T|},\quad R=\frac{|\hat T\cap T|}{|T|},\quad F_1=\frac{2PR}{P+R}
$$

- $y$: the target string of linearized triples (or JSON), trained with teacher forcing
- $\hat T,\;T$: predicted and gold triple sets; **strict** matching needs head span, tail span, (entity types) and relation all correct

Instruction-tuned LLMs do the same zero- or few-shot: give the schema and ask for JSON. One decoder pass handles all records and can resolve "she" from context, but outputs may not parse, may name entities that are not in the text, and are harder to calibrate than a classifier's probabilities (see **Hallucination detection**).

**Benchmarks**

- **ACE 2005:** news, broadcast and web text; 7 entity types, 6 relation types (18 subtypes), 33 event subtypes
- **TACRED:** Zhang et al. (2017): 106,264 sentence-level examples, 41 relations plus `no_relation` (about 79.5% of examples), built from TAC KBP slot filling
- **Metric:** micro P/R/F1 over triples; TACRED excludes `no_relation` from the positives; end-to-end systems report strict (types and boundaries) or boundaries-only scores, which are not comparable

> Figure: A pipeline relation extractor scored with strict triple F1. At the default τ = 0.50 with gold entities, 4 triples are emitted and 3 are correct, so P = R = F1 = 0.750: (Pierre Curie, born_in, Warsaw) at p = 0.621 is a false positive and the true (Pierre Curie, employee_of, University of Paris) at 0.473 falls below τ. Lowering τ to 0.45 gives F1 0.889 (P 0.8, R 1.0); raising it to 0.65 gives P 1.0, R 0.5, F1 0.667. Switch the NER stage: one boundary error ("Paris") drops F1 at τ = 0.50 to 0.500, and missing "Pierre Curie" caps recall at 0.5.

## Worked example: scoring one document

Softmax over (no_relation, born_in, employee_of, spouse) for each candidate pair, τ = 0.50, gold entities

| Pair | Argmax (p) | Emitted? | Gold | Result |
| --- | --- | --- | --- | --- |
| Marie Curie → Warsaw | born_in (0.861) | yes | born_in | TP |
| Marie Curie → University of Paris | employee_of (0.827) | yes | employee_of | TP |
| Marie Curie → Pierre Curie | spouse (0.619) | yes | spouse | TP |
| Marie Curie → 1895 | no_relation (0.635) | no | — | — |
| Pierre Curie → Warsaw | born_in (0.621) | yes | — | FP |
| Pierre Curie → University of Paris | employee_of (0.473) | no, below τ | employee_of | FN |
| Pierre Curie → 1895 | no_relation (0.684) | no | — | — |

$P=3/4=0.75$, $R=3/4=0.75$, $F_1=0.75$. Five of the seven pairs have a real-relation argmax, but only four have a gold relation; with longer documents the number of pairs grows quadratically while gold triples grow roughly linearly, so the `no_relation` class dominates.

```python
def parse_linearized(text):
    """Parse '<triplet> head <subj> tail <obj> relation ...' into (head, relation, tail) triples."""
    triples = set()
    for chunk in text.split("<triplet>")[1:]:
        head, rest = chunk.split("<subj>", 1)
        for pair in rest.split("<subj>"):           # one head can have several (tail, relation) pairs
            if "<obj>" not in pair:
                continue
            tail, rel = pair.split("<obj>", 1)
            triples.add((head.strip(), rel.strip(), tail.strip()))
    return triples

def strict_prf(pred, gold):
    tp = len(pred & gold)                            # exact match on head, relation and tail strings
    p = tp / len(pred) if pred else 0.0
    r = tp / len(gold) if gold else 0.0
    return p, r, (2 * p * r / (p + r) if p + r else 0.0)

generated = ("<triplet> Marie Curie <subj> Warsaw <obj> place of birth <subj> University of Paris <obj> employer "
             "<subj> Pierre Curie <obj> spouse <triplet> Pierre Curie <subj> Warsaw <obj> place of birth")
gold = {("Marie Curie", "place of birth", "Warsaw"), ("Marie Curie", "employer", "University of Paris"),
        ("Pierre Curie", "employer", "University of Paris"), ("Marie Curie", "spouse", "Pierre Curie")}
pred = parse_linearized(generated)
print(sorted(pred))
print("P=%.2f R=%.2f F1=%.2f" % strict_prf(pred, gold))   # P=0.75 R=0.75 F1=0.75
```

Parse a REBEL-style linearized output and score it with strict triple F1 (pure Python, runs as is; the string stands in for model output)

> **Compare only like-for-like scores:** End-to-end relation extraction papers differ in whether a triple needs correct entity **types** as well as boundaries, whether symmetric relations count in both directions, and whether gold entities are given. A model evaluated with gold entities (as on TACRED) is solving a much easier problem than one that must find them, as the "missed entity" setting in the figure shows.

**Strengths**

- Pipeline: each stage is simple, testable and calibrated; a strong pipeline matches joint models (Zhong and Chen, 2021)
- Generative: one model and one pass for entities, relations and events; new schemas need only new target strings or a prompt

**Limitations**

- Pipeline: NER errors propagate; one forward pass per candidate pair, quadratic in the number of entities
- Generative: output may be malformed or name entities absent from the text; no per-triple probability unless computed from token scores

**Question:** On TACRED about 79.5% of examples are no_relation. Why is micro-F1 computed with no_relation excluded from the positive classes?

- A. no_relation is never predicted
- B. A model that always predicts no_relation would otherwise look strong, with about 79.5% accuracy
- C. It makes the scores higher
- D. no_relation examples are removed from the test set

**Answer:** B. A model that always predicts no_relation would otherwise look strong, with about 79.5% accuracy — Accuracy counts every correct no_relation, so the trivial predictor scores about 79.5%. With no_relation excluded, that predictor has no true positives, so precision, recall and F1 are all 0. The metric then measures only the relations that matter.

**Question:** NER returns "Paris" instead of "University of Paris", and the classifier then correctly says employee_of. How does strict triple scoring count (Marie Curie, employee_of, Paris)?

- A. True positive: the relation is right
- B. Half credit
- C. A false positive, and the gold triple is a false negative
- D. It is ignored

**Answer:** C. A false positive, and the gold triple is a false negative — Strict matching requires the tail span to equal the gold one, so the predicted triple matches nothing (FP) and the gold triple is not found (FN). In the figure's "boundary error" setting, this happens to both employee_of triples.

**Key points**

- IE outputs records under a schema: entities, (head, relation, tail) triples, events (trigger + role arguments) and slots.
- Pipeline RE: NER, then $\operatorname{softmax}(W[h_{\langle E1\rangle};h_{\langle E2\rangle}])$ over relations plus no_relation for every marked pair.
- Errors propagate: a missed or mis-bounded entity makes every triple that contains it wrong.
- Generative IE (REBEL, TANL, UIE, prompted LLMs) writes triples as text; parse and validate before scoring.
- Report strict triple-level micro F1, and state whether gold entities were given.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
