Definition
Information extraction (IE) is the task of turning unstructured text into structured records with a fixed schema: entities (see Named entity recognition (NER)), relations between them as (head, relation, tail) triples, events (a trigger word plus arguments in named roles), and slots of a template about a given entity. The task was shaped by the MUC evaluations (1987–1997) and the ACE program; ACE 2005 is still a standard benchmark.
Input: a sentence or document. Output: a set of records. From "Marie Curie was born in Warsaw" a system should produce (Marie Curie, born_in, Warsaw). The hard parts: the same relation is phrased in many ways, most entity pairs in a text are not related, arguments may be pronouns ("she"), and the relation may span sentences.
Before deep learning, relation extraction used hand-written patterns and feature-based or kernel classifiers over words, entity types and dependency paths. Distant supervision (Mintz et al., 2009) created training data by labelling every sentence that mentions a Freebase entity pair with that pair's relation, which is cheap but noisy.
The subtasks
| Subtask | Output | Typical neural model |
|---|---|---|
| Relation extraction | (head, relation, tail) or no_relation per entity pair | encoder + classifier on the two entity representations |
| Event extraction | trigger span + type; arguments with roles (Person, Time, Place, …) | trigger tagging, then argument classification per (trigger, entity) |
| Slot filling | values for a fixed set of slots: of a query entity (KBP), or of an utterance (dialogue, e.g. ATIS) | relation classifier per candidate (KBP); BIO tagging (dialogue) |
| Open IE | (arg1, relation phrase, arg2) with no fixed schema | TextRunner (Banko et al., 2007), ReVerb (Fader et al., 2011); later neural taggers and seq2seq |
Pipeline relation extraction with entity markers
The pipeline runs NER first, then forms every candidate pair and classifies it. The standard input inserts marker tokens around the two entities, e.g. [E1] Marie Curie [/E1] was born in [E2] Warsaw [/E2]. Baldini Soares et al. (2019, "Matching the Blanks") found that concatenating the encoder states at the two start markers works best:
- final encoder states (e.g. BERT) at the start markers of head and tail
- the schema's relation types;
no_relationabsorbs the many unrelated pairs - a linear layer trained with cross-entropy
Each pair needs a forward pass, and NER errors propagate: a pair containing a missed or mis-bounded entity can never be correct. If each entity were found exactly with probability 0.9, independently, a pair would survive with . Joint models predict entities and relations together, e.g. Miwa and Bansal (2016) with LSTMs over sequences and dependency trees. Zhong and Chen (2021, "A Frustratingly Easy Approach for Entity and Relation Extraction") then showed that a pipeline with separate encoders and typed markers matches or beats joint models on ACE and SciERC.
Generative IE
A sequence-to-sequence model (see BART, T5) can write the records as text: train where is a linearized list of triples, then parse . REBEL (Huguet Cabot and Navigli, 2021) fine-tunes BART-large on the format <triplet> head <subj> tail <obj> relation. TANL (Paolini et al., 2021) writes augmented copies of the input, and UIE (Lu et al., 2022) uses one structured language for entities, relations, events and sentiment.
- the target string of linearized triples (or JSON), trained with teacher forcing
- predicted and gold triple sets; strict matching needs head span, tail span, (entity types) and relation all correct
Instruction-tuned LLMs do the same zero- or few-shot: give the schema and ask for JSON. One decoder pass handles all records and can resolve "she" from context, but outputs may not parse, may name entities that are not in the text, and are harder to calibrate than a classifier's probabilities (see Hallucination detection).
Benchmarks
- ACE 2005
- news, broadcast and web text; 7 entity types, 6 relation types (18 subtypes), 33 event subtypes
- TACRED
- Zhang et al. (2017): 106,264 sentence-level examples, 41 relations plus
no_relation(about 79.5% of examples), built from TAC KBP slot filling - Metric
- micro P/R/F1 over triples; TACRED excludes
no_relationfrom the positives; end-to-end systems report strict (types and boundaries) or boundaries-only scores, which are not comparable
Worked example: scoring one document
| Pair | Argmax (p) | Emitted? | Gold | Result |
|---|---|---|---|---|
| Marie Curie → Warsaw | born_in (0.861) | yes | born_in | TP |
| Marie Curie → University of Paris | employee_of (0.827) | yes | employee_of | TP |
| Marie Curie → Pierre Curie | spouse (0.619) | yes | spouse | TP |
| Marie Curie → 1895 | no_relation (0.635) | no | — | — |
| Pierre Curie → Warsaw | born_in (0.621) | yes | — | FP |
| Pierre Curie → University of Paris | employee_of (0.473) | no, below τ | employee_of | FN |
| Pierre Curie → 1895 | no_relation (0.684) | no | — | — |
, , . Five of the seven pairs have a real-relation argmax, but only four have a gold relation; with longer documents the number of pairs grows quadratically while gold triples grow roughly linearly, so the no_relation class dominates.
1def parse_linearized(text):2 """Parse '<triplet> head <subj> tail <obj> relation ...' into (head, relation, tail) triples."""3 triples = set()4 for chunk in text.split("<triplet>")[1:]:5 head, rest = chunk.split("<subj>", 1)6 for pair in rest.split("<subj>"): # one head can have several (tail, relation) pairs7 if "<obj>" not in pair:8 continue9 tail, rel = pair.split("<obj>", 1)10 triples.add((head.strip(), rel.strip(), tail.strip()))11 return triples1213def strict_prf(pred, gold):14 tp = len(pred & gold) # exact match on head, relation and tail strings15 p = tp / len(pred) if pred else 0.016 r = tp / len(gold) if gold else 0.017 return p, r, (2 * p * r / (p + r) if p + r else 0.0)1819generated = ("<triplet> Marie Curie <subj> Warsaw <obj> place of birth <subj> University of Paris <obj> employer "20 "<subj> Pierre Curie <obj> spouse <triplet> Pierre Curie <subj> Warsaw <obj> place of birth")21gold = {("Marie Curie", "place of birth", "Warsaw"), ("Marie Curie", "employer", "University of Paris"),22 ("Pierre Curie", "employer", "University of Paris"), ("Marie Curie", "spouse", "Pierre Curie")}23pred = parse_linearized(generated)24print(sorted(pred))25print("P=%.2f R=%.2f F1=%.2f" % strict_prf(pred, gold)) # P=0.75 R=0.75 F1=0.75Use when
- +Pipeline: each stage is simple, testable and calibrated; a strong pipeline matches joint models (Zhong and Chen, 2021)
- +Generative: one model and one pass for entities, relations and events; new schemas need only new target strings or a prompt
Avoid when
- −Pipeline: NER errors propagate; one forward pass per candidate pair, quadratic in the number of entities
- −Generative: output may be malformed or name entities absent from the text; no per-triple probability unless computed from token scores
Check yourself
On TACRED about 79.5% of examples are no_relation. Why is micro-F1 computed with no_relation excluded from the positive classes?
Check yourself
NER returns "Paris" instead of "University of Paris", and the classifier then correctly says employee_of. How does strict triple scoring count (Marie Curie, employee_of, Paris)?
Remember
- ✓IE outputs records under a schema: entities, (head, relation, tail) triples, events (trigger + role arguments) and slots.
- ✓Pipeline RE: NER, then over relations plus no_relation for every marked pair.
- ✓Errors propagate: a missed or mis-bounded entity makes every triple that contains it wrong.
- ✓Generative IE (REBEL, TANL, UIE, prompted LLMs) writes triples as text; parse and validate before scoring.
- ✓Report strict triple-level micro F1, and state whether gold entities were given.