Information extraction

Information extraction turns text into typed records (entities, relation triples, events, slots); pipelines classify marked entity pairs, generative models such as REBEL write the triples as text, and both are scored by strict triple-level F1.

8 min read613 wordscore0 viewsupdated 2026-10-15

Definition

Information extraction (IE) is the task of turning unstructured text into structured records with a fixed schema: entities (see Named entity recognition (NER)), relations between them as (head, relation, tail) triples, events (a trigger word plus arguments in named roles), and slots of a template about a given entity. The task was shaped by the MUC evaluations (1987–1997) and the ACE program; ACE 2005 is still a standard benchmark.

Input: a sentence or document. Output: a set of records. From "Marie Curie was born in Warsaw" a system should produce (Marie Curie, born_in, Warsaw). The hard parts: the same relation is phrased in many ways, most entity pairs in a text are not related, arguments may be pronouns ("she"), and the relation may span sentences.

Before deep learning, relation extraction used hand-written patterns and feature-based or kernel classifiers over words, entity types and dependency paths. Distant supervision (Mintz et al., 2009) created training data by labelling every sentence that mentions a Freebase entity pair with that pair's relation, which is cheap but noisy.

unstructured text in, typed records outMarie Curie was born in Warsaw. She worked at the University of Paris withPierre Curie, whom she married in 1895.entities (NER)Marie Curie PER · Warsaw LOCUniversity of Paris ORGPierre Curie PER · 1895 DATErelations (head, relation, tail)(Marie Curie, born_in, Warsaw)(Marie Curie, employee_of, Univ. of Paris)(Pierre Curie, employee_of, Univ. of Paris)(Marie Curie, spouse, Pierre Curie)event: Life:Marry (ACE 2005)trigger “married”Person Marie Curie (“she”)Person Pierre Curie (“whom”)Time 1895slot filling for query “Marie Curie”per:city_of_birth = Warsawper:employee_of = University of Parisper:spouse = Pierre Curie(TAC KBP / TACRED relation names)how the relation triples are producedA. pipeline: NER, then classify each pair[E1] Marie Curie [/E1] was born in[E2] Warsaw [/E2] . She worked ...encoder → [h_[E1] ; h_[E2]] → softmaxover relations + no_relation → born_inone pass per pair; NER errors propagateB. generative: write the triples as texttext → seq2seq (BART/T5) or LLM →<triplet> Marie Curie <subj> Warsaw<obj> place of birth <subj> ...parse → triples (REBEL-style), or JSONone pass for all; output may not parse
fig.Four IE outputs from one sentence: entities, relation triples, an ACE-style Life:Marry event (trigger "married" with Person and Time arguments), and template slots for the query entity "Marie Curie". Below, the two ways to produce the triples: a pipeline that marks each entity pair and classifies it, or a sequence-to-sequence model that writes the triples as a linearized string to be parsed.

The subtasks

SubtaskOutputTypical neural model
Relation extraction(head, relation, tail) or no_relation per entity pairencoder + classifier on the two entity representations
Event extractiontrigger span + type; arguments with roles (Person, Time, Place, …)trigger tagging, then argument classification per (trigger, entity)
Slot fillingvalues for a fixed set of slots: of a query entity (KBP), or of an utterance (dialogue, e.g. ATIS)relation classifier per candidate (KBP); BIO tagging (dialogue)
Open IE(arg1, relation phrase, arg2) with no fixed schemaTextRunner (Banko et al., 2007), ReVerb (Fader et al., 2011); later neural taggers and seq2seq

Pipeline relation extraction with entity markers

The pipeline runs NER first, then forms every candidate pair and classifies it. The standard input inserts marker tokens around the two entities, e.g. [E1] Marie Curie [/E1] was born in [E2] Warsaw [/E2]. Baldini Soares et al. (2019, "Matching the Blanks") found that concatenating the encoder states at the two start markers works best:

p(r∣x,e1,e2)=softmax⁡(W [ h⟨E1⟩; h⟨E2⟩ ]+b)r,r∈R∪{no_relation}p(r\mid x,e_1,e_2)=\operatorname{softmax}\big(W\,[\,h_{\langle E1\rangle};\,h_{\langle E2\rangle}\,]+b\big)_r,\qquad r\in\mathcal R\cup\{\texttt{no\_relation}\}
h⟨E1⟩,h⟨E2⟩h_{\langle E1\rangle},h_{\langle E2\rangle}
final encoder states (e.g. BERT) at the start markers of head and tail
R\mathcal R
the schema's relation types; no_relation absorbs the many unrelated pairs
W∈R(∣R∣+1)×2HW\in\mathbb R^{(|\mathcal R|+1)\times2H}
a linear layer trained with cross-entropy

Each pair needs a forward pass, and NER errors propagate: a pair containing a missed or mis-bounded entity can never be correct. If each entity were found exactly with probability 0.9, independently, a pair would survive with 0.92=0.810.9^2=0.81. Joint models predict entities and relations together, e.g. Miwa and Bansal (2016) with LSTMs over sequences and dependency trees. Zhong and Chen (2021, "A Frustratingly Easy Approach for Entity and Relation Extraction") then showed that a pipeline with separate encoders and typed markers matches or beats joint models on ACE and SciERC.

Generative IE

A sequence-to-sequence model (see BART, T5) can write the records as text: train p(y∣x)p(y\mid x) where yy is a linearized list of triples, then parse yy. REBEL (Huguet Cabot and Navigli, 2021) fine-tunes BART-large on the format <triplet> head <subj> tail <obj> relation. TANL (Paolini et al., 2021) writes augmented copies of the input, and UIE (Lu et al., 2022) uses one structured language for entities, relations, events and sentiment.

p(y∣x)=∏t=1∣y∣p(yt∣y<t,x),P=∣T^∩T∣∣T^∣,R=∣T^∩T∣∣T∣,F1=2PRP+Rp(y\mid x)=\prod_{t=1}^{|y|}p\big(y_t\mid y_{<t},x\big),\qquad P=\frac{|\hat T\cap T|}{|\hat T|},\quad R=\frac{|\hat T\cap T|}{|T|},\quad F_1=\frac{2PR}{P+R}
yy
the target string of linearized triples (or JSON), trained with teacher forcing
T^,  T\hat T,\;T
predicted and gold triple sets; strict matching needs head span, tail span, (entity types) and relation all correct

Instruction-tuned LLMs do the same zero- or few-shot: give the schema and ask for JSON. One decoder pass handles all records and can resolve "she" from context, but outputs may not parse, may name entities that are not in the text, and are harder to calibrate than a classifier's probabilities (see Hallucination detection).

Benchmarks

ACE 2005
news, broadcast and web text; 7 entity types, 6 relation types (18 subtypes), 33 event subtypes
TACRED
Zhang et al. (2017): 106,264 sentence-level examples, 41 relations plus no_relation (about 79.5% of examples), built from TAC KBP slot filling
Metric
micro P/R/F1 over triples; TACRED excludes no_relation from the positives; end-to-end systems report strict (types and boundaries) or boundaries-only scores, which are not comparable
“Marie Curie was born in Warsaw. She worked at the University of Pariswith Pierre Curie, whom she married in 1895.”stage 1, NER (gold entities):Marie Curie PERWarsaw LOCUniversity of Paris ORGPierre Curie PER1895 DATEstage 2, relation classifier per (PER, entity) pair · emit argmax ≠ no_relation if p ≥ τ = 0.50pair (head → tail)p over 4 classespredictiongoldresultMarie → Warsawborn_in .861born_inTPMarie → University of Parisemployee_of .827employee_ofTPMarie → Pierre Curiespouse .619spouseTPMarie → 1895no_relation .635——Pierre → Warsawborn_in .621—FPPierre → University of Parisemployee_of .473 <τemployee_ofFNPierre → 1895no_relation .684——■ born_in ■ employee_of ■ spouse ■ no_relation · dotted tick = τpredicted triples4correct (strict match)3 of 4 goldprecision0.750recall0.750F10.750with gold entities, only the classifier and τ limit the scoreF1 vs τ (this NER, ┄ gold NER)00.510.30.50.70.9strict = head, relation and tail strings all equal a gold triple; no partial credit for “Paris”
NER stage
fig.A pipeline relation extractor scored with strict triple F1. At the default τ = 0.50 with gold entities, 4 triples are emitted and 3 are correct, so P = R = F1 = 0.750: (Pierre Curie, born_in, Warsaw) at p = 0.621 is a false positive and the true (Pierre Curie, employee_of, University of Paris) at 0.473 falls below τ. Lowering τ to 0.45 gives F1 0.889 (P 0.8, R 1.0); raising it to 0.65 gives P 1.0, R 0.5, F1 0.667. Switch the NER stage: one boundary error ("Paris") drops F1 at τ = 0.50 to 0.500, and missing "Pierre Curie" caps recall at 0.5.

Worked example: scoring one document

PairArgmax (p)Emitted?GoldResult
Marie Curie → Warsawborn_in (0.861)yesborn_inTP
Marie Curie → University of Parisemployee_of (0.827)yesemployee_ofTP
Marie Curie → Pierre Curiespouse (0.619)yesspouseTP
Marie Curie → 1895no_relation (0.635)no——
Pierre Curie → Warsawborn_in (0.621)yes—FP
Pierre Curie → University of Parisemployee_of (0.473)no, below τemployee_ofFN
Pierre Curie → 1895no_relation (0.684)no——
Softmax over (no_relation, born_in, employee_of, spouse) for each candidate pair, τ = 0.50, gold entities

P=3/4=0.75P=3/4=0.75, R=3/4=0.75R=3/4=0.75, F1=0.75F_1=0.75. Five of the seven pairs have a real-relation argmax, but only four have a gold relation; with longer documents the number of pairs grows quadratically while gold triples grow roughly linearly, so the no_relation class dominates.

python
1def parse_linearized(text):
2 """Parse '<triplet> head <subj> tail <obj> relation ...' into (head, relation, tail) triples."""
3 triples = set()
4 for chunk in text.split("<triplet>")[1:]:
5 head, rest = chunk.split("<subj>", 1)
6 for pair in rest.split("<subj>"): # one head can have several (tail, relation) pairs
7 if "<obj>" not in pair:
8 continue
9 tail, rel = pair.split("<obj>", 1)
10 triples.add((head.strip(), rel.strip(), tail.strip()))
11 return triples
12
13def strict_prf(pred, gold):
14 tp = len(pred & gold) # exact match on head, relation and tail strings
15 p = tp / len(pred) if pred else 0.0
16 r = tp / len(gold) if gold else 0.0
17 return p, r, (2 * p * r / (p + r) if p + r else 0.0)
18
19generated = ("<triplet> Marie Curie <subj> Warsaw <obj> place of birth <subj> University of Paris <obj> employer "
20 "<subj> Pierre Curie <obj> spouse <triplet> Pierre Curie <subj> Warsaw <obj> place of birth")
21gold = {("Marie Curie", "place of birth", "Warsaw"), ("Marie Curie", "employer", "University of Paris"),
22 ("Pierre Curie", "employer", "University of Paris"), ("Marie Curie", "spouse", "Pierre Curie")}
23pred = parse_linearized(generated)
24print(sorted(pred))
25print("P=%.2f R=%.2f F1=%.2f" % strict_prf(pred, gold)) # P=0.75 R=0.75 F1=0.75
Parse a REBEL-style linearized output and score it with strict triple F1 (pure Python, runs as is; the string stands in for model output)

Use when

  • +Pipeline: each stage is simple, testable and calibrated; a strong pipeline matches joint models (Zhong and Chen, 2021)
  • +Generative: one model and one pass for entities, relations and events; new schemas need only new target strings or a prompt

Avoid when

  • −Pipeline: NER errors propagate; one forward pass per candidate pair, quadratic in the number of entities
  • −Generative: output may be malformed or name entities absent from the text; no per-triple probability unless computed from token scores

Check yourself

On TACRED about 79.5% of examples are no_relation. Why is micro-F1 computed with no_relation excluded from the positive classes?

Check yourself

NER returns "Paris" instead of "University of Paris", and the classifier then correctly says employee_of. How does strict triple scoring count (Marie Curie, employee_of, Paris)?

Remember

  • ✓IE outputs records under a schema: entities, (head, relation, tail) triples, events (trigger + role arguments) and slots.
  • ✓Pipeline RE: NER, then softmax⁡(W[h⟨E1⟩;h⟨E2⟩])\operatorname{softmax}(W[h_{\langle E1\rangle};h_{\langle E2\rangle}]) over relations plus no_relation for every marked pair.
  • ✓Errors propagate: a missed or mis-bounded entity makes every triple that contains it wrong.
  • ✓Generative IE (REBEL, TANL, UIE, prompted LLMs) writes triples as text; parse and validate before scoring.
  • ✓Report strict triple-level micro F1, and state whether gold entities were given.
← back to Deep Learning roadmap