Knowledge graph embeddings

Entities and relations become vectors scored by a geometric function (TransE translation, DistMult/ComplEx products, RotatE rotation) that fixes which relation patterns fit; judged by filtered MRR on FB15k-237 and WN18RR.

8 min read742 wordsadvanced0 viewsupdated 2026-10-21

Definition

Knowledge graph embedding assigns a vector to every entity and every relation of a knowledge graph, a set of triples (h,r,t)(h,r,t) such as (Paris, capital_of, France), together with a score f(h,r,t)f(h,r,t) that should be high for true triples. The main models are TransE (Bordes, Usunier, García-Durán, Weston and Yakhnenko, NeurIPS 2013), DistMult (Yang, Yih, He, Gao and Deng, ICLR 2015), ComplEx (Trouillon, Welbl, Riedel, Gaussier and Bouchard, ICML 2016) and RotatE (Sun, Deng, Nie and Tang, ICLR 2019).

The task is knowledge-graph completion. Freebase and Wikidata hold millions of facts but miss many more: a person's nationality or a film's director is often absent. Given a query (h,r,?)(h,r,?) or (?,r,t)(?,r,t), the model ranks every entity as a candidate answer. This is Link prediction on a multi-relational graph: an edge type matters, and the same pair can be linked by several relations.

Two things make it hard. The graph is incomplete, so a missing triple is not necessarily false. Relations also behave differently: "borders" is symmetric, "parent_of" is antisymmetric, "has_part" is the inverse of "part_of", and "born_in" followed by "city_of" gives "nationality". Before learned embeddings, systems mined such rules explicitly. RESCAL (Nickel, Tresp and Kriegel, ICML 2011) factorised the adjacency tensor with a full matrix per relation, f=h⊤Mrtf=h^\top M_r t.

Score functions

Knowledge graph embedding: score every triple (h, r, t) with a geometric function1 · triples (h, r, t)capital_ofcapital_ofbordersParisFranceMadridSpain2 · look up vectorsentities E|E| × drelations R|R| × df(h, r, t)one scalarFB15k-237: 14,541entities, 237 rels3 · score function and the relation patterns it can expressTransE (2013)−‖h + r − t‖h, r, t ∈ ℝ^d✗ symmetry✓ antisymmetry✓ inversion✓ compositionDistMult (2015)Σ h_i r_i t_ih, r, t ∈ ℝ^d✓ symmetry✗ antisymmetry✗ inversion✗ compositionComplEx (2016)Re Σ h_i r_i t̄_ih, r, t ∈ ℂ^d✓ symmetry✓ antisymmetry✓ inversion✗ compositionRotatE (2019)−‖h ∘ r − t‖|r_i| = 1 in ℂ^d✓ symmetry✓ antisymmetry✓ inversion✓ composition4 · train against corrupted triples(Paris, capital_of, Spain) is a negativeloss: [γ + d(pos) − d(neg)]₊ or −log σ5 · rank every entity: (Madrid, capital_of, ?)drop other true answers (filtered)MRR = mean 1/rank, Hits@k = P(rank ≤ k)
fig.The pipeline. (1) Triples of a small knowledge graph. (2) Each entity and relation indexes a row of an embedding table. (3) A score function turns the three vectors into one number; which one decides the relation patterns the model can express. (4) Training pushes true triples above corrupted ones. (5) Evaluation ranks every entity for a query, removes the other known answers, and reports MRR and Hits@k.
TransE: f=−∥h+r−t∥1 or 2DistMult: f=∑ihiritiComplEx: f=Re⁡∑ihiritˉiRotatE: f=−∥h∘r−t∥,  ri=eiθi\begin{aligned}\text{TransE: }&f=-\lVert h+r-t\rVert_{1\text{ or }2} &\quad \text{DistMult: }&f=\textstyle\sum_i h_i r_i t_i\\ \text{ComplEx: }&f=\operatorname{Re}\textstyle\sum_i h_i r_i \bar t_i &\quad \text{RotatE: }&f=-\lVert h\circ r-t\rVert,\ \ r_i=e^{i\theta_i}\end{aligned}
h, r, th,\ r,\ t
head, relation and tail embeddings; real in TransE and DistMult, complex (Cd\mathbb{C}^d) in ComplEx and RotatE
tˉ\bar t
complex conjugate; it makes the ComplEx score change when hh and tt swap
∘\circ
element-wise product; with ∣ri∣=1|r_i|=1 each coordinate of hh is rotated by θi\theta_i

TransE models a relation as a translation: h+r≈th+r\approx t, the same geometry as the Word2Vec analogy king−man+woman≈queen\textit{king}-\textit{man}+\textit{woman}\approx\textit{queen}. It keeps entity vectors at unit norm. A symmetric relation needs h+r=th+r=t and t+r=ht+r=h, so r=0r=0 and h=th=t. A one-to-many relation forces all tails onto one point. TransH (Wang et al., AAAI 2014) and TransR (Lin et al., AAAI 2015) project entities into a relation-specific space to ease this.

DistMult is RESCAL with a diagonal MrM_r. It has dd parameters per relation instead of d2d^2, but its score is symmetric in hh and tt, so it predicts (France, capital_of, Paris) as readily as the true triple. ComplEx moves to complex vectors and conjugates the tail. A real rr gives a symmetric relation and an imaginary rr an antisymmetric one. RotatE constrains every rir_i to the unit circle. Symmetry is θi∈{0,π}\theta_i\in\{0,\pi\}, an inverse relation is the conjugate rotation, and composition adds angles.

ModelSymmetryAntisymmetryInversionComposition
TransE✗✓✓✓
DistMult✓✗✗✗
ComplEx✓✓✓✗
RotatE✓✓✓✓
Relation patterns each score function can express (Sun et al., 2019, Table 2)

Training

Only positive triples are given. Negatives are made by corrupting the head or the tail of a true triple with a random entity. TransE minimises the margin ranking loss ∑[γ+d(h,r,t)−d(h′,r,t′)]+\sum[\gamma+d(h,r,t)-d(h',r,t')]_+. RotatE uses a log-sigmoid loss with self-adversarial sampling, which weights each negative by how plausible the current model finds it:

L=−log⁡σ(γ−dr(h,t))−∑i=1np(hi′,r,ti′) log⁡σ(dr(hi′,ti′)−γ),p(hj′,r,tj′)=exp⁡αf(hj′,r,tj′)∑iexp⁡αf(hi′,r,ti′)\mathcal{L}=-\log\sigma\big(\gamma-d_r(h,t)\big)-\sum_{i=1}^{n}p(h'_i,r,t'_i)\,\log\sigma\big(d_r(h'_i,t'_i)-\gamma\big),\qquad p(h'_j,r,t'_j)=\frac{\exp\alpha f(h'_j,r,t'_j)}{\sum_i\exp\alpha f(h'_i,r,t'_i)}
dr(h,t)d_r(h,t)
the distance ∥h∘r−t∥\lVert h\circ r-t\rVert; the score is −dr-d_r
γ\gamma
a fixed margin
α\alpha
sampling temperature; the weights pp are treated as constants (no gradient)

Easy negatives such as (Paris, capital_of, Banana) give almost no gradient after a few epochs. Weighting by plausibility spends the update on hard ones like (Paris, capital_of, Belgium).

Evaluation: filtered ranks

MRR=1∣Q∣∑q∈Q1rank⁡q,Hits@k=1∣Q∣∑q∈Q1[rank⁡q≤k]\text{MRR}=\frac{1}{|Q|}\sum_{q\in Q}\frac{1}{\operatorname{rank}_q},\qquad \text{Hits@}k=\frac{1}{|Q|}\sum_{q\in Q}\mathbb{1}[\operatorname{rank}_q\le k]
QQ
a tail query (h,r,?)(h,r,?) and a head query (?,r,t)(?,r,t) for every test triple
rank⁡q\operatorname{rank}_q
position of the true entity among all entities, after removing every other entity that forms a known true triple (train, validation or test): the filtered setting of Bordes et al.

Without filtering, a model that ranks Lyon first for (?, located_in, France) is penalised even though Lyon is a correct answer. Filtered MRR is the headline number in most papers, usually with Hits@1, @3 and @10.

QueryPredictionNearest entities (distance)Raw rankFiltered rank
(Paris, located_in, ?) → Franceh + r = (1, 1)France 0.141, Rome 0.86011
(Madrid, located_in, ?) → Spainh + r = (1, −1)Spain 0.224, Italy 0.89411
(Rome, located_in, ?) → Italyh + r = (1.3, 0.5)France 0.447, Italy 0.70722
(?, located_in, France) → Parist − r = (0.1, 0.9)Lyon 0.100, Paris 0.14121 (Lyon is known)
Metrics over the 4 queriesMRR 0.75, Hits@1 0.50MRR 0.875, Hits@1 0.75
Worked example: TransE in 2-D with r = located_in = (1, 0). Entities: Paris (0, 1), Lyon (0.1, 0.8), Madrid (0, −1), Rome (0.3, 0.5), France (1.1, 0.9), Spain (0.9, −1.2), Italy (1.4, −0.2). (Lyon, located_in, France) is a training triple.

Benchmarks and results

FB15k-237
Toutanova and Chen (2015): 14,541 entities, 237 relations; 272,115 / 17,535 / 20,466 train / valid / test triples
WN18RR
Dettmers et al. (AAAI 2018): 40,943 WordNet synsets, 11 relations; 86,835 / 3,034 / 3,134 triples
Why the new splits
in the older FB15k and WN18, many test triples are inverses of training triples, so a simple inverse-relation rule scored near the top; the -237 and RR versions remove such relations
DistMult / ComplEx
filtered MRR 0.241 / 0.247 on FB15k-237 and 0.43 / 0.44 on WN18RR, as reported by Dettmers et al. (2018)
RotatE
filtered MRR 0.338, Hits@10 0.533 on FB15k-237; MRR 0.476, Hits@10 0.571 on WN18RR (Sun et al., 2019, with self-adversarial sampling)
TransE: score = −‖h + r − t‖ · epoch 200, margin loss 0.1239 training triples · Ann, Ben → Cal, Dee → Eve, Fay (manages) · peers in each rowAnnBenCalDeeEveFay× = prediction for (Eve, peer_of, ?) · dashed = true tail(Eve, peer_of, ?) → Fay, held outFay0.046Eve0.065Dee0.853Cal0.864Ann1.573Ben1.574filtered rank of Fay: 1train MRR 0.722 · Hits@1 44%manages MRR 0.750 · peer_of MRR 0.700held-out (Eve, peer_of, Fay): ranks 1 / 1(tail / head query, filtered, 6 candidates)‖r_manages‖ = 0.815 · ‖r_peer_of‖ = 0.065symmetry forces r ≈ 0 and h ≈ t:peers collapse onto one point
model
query
fig.TransE and RotatE trained in the browser (margin 0.5, SGD lr 0.05, 2 negatives per triple, entities on the unit circle) on 9 triples: Ann → Cal → Eve and Ben → Dee → Fay under manages, and peer_of between Ann–Ben and Cal–Dee in both directions and Fay → Eve. The test triple (Eve, peer_of, Fay) is held out. Default (TransE, epoch 200): ‖r_peer_of‖ = 0.065 against ‖r_manages‖ = 0.815, and the partners collapse (Ann 104.4°, Ben 104.5°). Fay at 0.046 beats Eve herself at 0.065, but at 13 of the 19 snapshots from epoch 20 Eve is nearer and the held-out rank is 2 (epoch 190: 0.101 vs 0.160). Train MRR 0.722, Hits@1 44%. Switch to RotatE: loss 0 from epoch 20, θ_peer_of = 182.9°, θ_manages = 239.7°, train MRR 1.000, and the held-out triple ranks first both ways (Fay 0.284, next Ann 0.940).

Connection to graph neural networks

All four models are shallow: the encoder is a lookup table, as in DeepWalk (see Graph representation learning). A GNN can replace it. The Relational GCN (Schlichtkrull et al., ESWC 2018) computes entity vectors by message passing with one weight matrix per relation and scores triples with a DistMult decoder; CompGCN (Vashishth et al., ICLR 2020) also embeds the relations and composes them with entities using TransE-, DistMult- or circular-correlation operators. Message passing lets an entity with few facts borrow evidence from its neighbours, at the cost of a far more expensive forward pass.

python
1import numpy as np
2
3# score functions: higher = more plausible
4transe = lambda h, r, t: -np.linalg.norm(h + r - t, axis=-1)
5distmult = lambda h, r, t: np.sum(h * r * t, axis=-1)
6complex_ = lambda h, r, t: np.real(np.sum(h * r * np.conj(t), axis=-1))
7rotate = lambda h, theta, t: -np.linalg.norm(h * np.exp(1j * theta) - t, axis=-1)
8
9rng = np.random.default_rng(0)
10d = 50
11h, t = rng.normal(size=(2, d))
12hc, tc = rng.normal(size=(2, d)) + 1j * rng.normal(size=(2, d))
13r = rng.normal(size=d)
14
15print(np.isclose(distmult(h, r, t), distmult(t, r, h))) # True: DistMult is always symmetric
16print(np.isclose(complex_(hc, r + 0j, tc), complex_(tc, r + 0j, hc))) # True: real r -> symmetric
17print(np.isclose(complex_(hc, 1j * r, tc), -complex_(tc, 1j * r, hc))) # True: imaginary r -> antisymmetric
18th = np.full(d, np.pi) # rotation by pi is its own inverse
19print(np.isclose(rotate(hc, th, -hc), 0), np.isclose(rotate(-hc, th, hc), 0)) # True True: h -> -h and back
20
21def filtered_rank(scores, truth, known):
22 """Rank of the true entity among all candidates, ignoring other known true answers."""
23 mask = np.ones_like(scores, dtype=bool)
24 mask[list(known - {truth})] = False
25 return 1 + int(np.sum(scores[mask] > scores[truth]))
26
27# the worked example: TransE in 2-D, r = located_in = (1, 0)
28names = ["Paris", "Lyon", "Madrid", "Rome", "France", "Spain", "Italy"]
29E = np.array([[0, 1], [0.1, 0.8], [0, -1], [0.3, 0.5], [1.1, 0.9], [0.9, -1.2], [1.4, -0.2]])
30loc = np.array([1.0, 0.0])
31ranks = [
32 filtered_rank(transe(E[0], loc, E), 4, set()), # (Paris, located_in, ?) -> France
33 filtered_rank(transe(E[2], loc, E), 5, set()), # (Madrid, located_in, ?) -> Spain
34 filtered_rank(transe(E[3], loc, E), 6, set()), # (Rome, located_in, ?) -> Italy
35 filtered_rank(transe(E, loc, E[4]), 0, {1}), # (?, located_in, France) -> Paris; Lyon is known
36]
37ranks = np.array(ranks)
38print(ranks, "MRR", (1 / ranks).mean(), "Hits@1", (ranks <= 1).mean()) # [1 1 2 1] MRR 0.875 Hits@1 0.75
The four score functions, their relation patterns, and filtered ranks on the worked example (numpy, runs as is)

Where it's used

Knowledge-base completion

Suggesting missing facts in Freebase-, Wikidata- or enterprise-scale graphs for human review.

Biomedical discovery

Ranking drug–target or drug–disease links in graphs of genes, proteins, compounds and diseases, e.g. for drug repurposing.

Recommendation and search

Item and user embeddings that respect typed relations (bought, belongs_to, made_by) and feed retrieval or ranking models.

Check yourself

Why can DistMult not model the relation parent_of correctly?

Check yourself

For (?, located_in, France), the true answer Paris is ranked 2nd, behind Lyon, which is also located in France and appears in the training set. What is the filtered rank?

Remember

  • ✓A KG embedding is a lookup table for entities and relations plus a score f(h,r,t)f(h,r,t); completion ranks all entities for (h,r,?)(h,r,?) and (?,r,t)(?,r,t).
  • ✓TransE: −∥h+r−t∥-\lVert h+r-t\rVert, translations; cannot express symmetry (r=0r=0 forces h=th=t) and struggles with one-to-many relations.
  • ✓DistMult: ∑hiriti\sum h_ir_it_i, cheap but always symmetric. ComplEx: Re⁡∑hiritˉi\operatorname{Re}\sum h_ir_i\bar t_i, adds antisymmetry and inversion.
  • ✓RotatE: −∥h∘r−t∥-\lVert h\circ r-t\rVert with ∣ri∣=1|r_i|=1; rotations give symmetry (θ=π\theta=\pi), inversion (conjugate) and composition (adding angles).
  • ✓Train with corrupted negatives (margin loss, or log-sigmoid with self-adversarial weights); report filtered MRR and Hits@k on FB15k-237 and WN18RR, not the leaky FB15k and WN18.
  • ✓Training protocol matters as much as the score function; GNN encoders such as R-GCN can replace the lookup table.
← back to Deep Learning roadmap