# Knowledge graph embeddings

> Entities and relations become vectors scored by a geometric function (TransE translation, DistMult/ComplEx products, RotatE rotation) that fixes which relation patterns fit; judged by filtered MRR on FB15k-237 and WN18RR.

Source: https://www.mlfoundry.in/learn/knowledge-graph-embeddings  
Section: Deep Learning › Graph Neural Networks (GNNs)  
Level: advanced · 8 min read · updated 2026-10-21

**Knowledge graph embedding:** assigns a vector to every entity and every relation of a knowledge graph, a set of triples $(h,r,t)$ such as (Paris, capital_of, France), together with a score $f(h,r,t)$ that should be high for true triples. The main models are TransE (Bordes, Usunier, García-Durán, Weston and Yakhnenko, NeurIPS 2013), DistMult (Yang, Yih, He, Gao and Deng, ICLR 2015), ComplEx (Trouillon, Welbl, Riedel, Gaussier and Bouchard, ICML 2016) and RotatE (Sun, Deng, Nie and Tang, ICLR 2019).

The task is knowledge-graph completion. Freebase and Wikidata hold millions of facts but miss many more: a person's nationality or a film's director is often absent. Given a query $(h,r,?)$ or $(?,r,t)$, the model ranks every entity as a candidate answer. This is **Link prediction** on a multi-relational graph: an edge type matters, and the same pair can be linked by several relations.

Two things make it hard. The graph is incomplete, so a missing triple is not necessarily false. Relations also behave differently: "borders" is symmetric, "parent_of" is antisymmetric, "has_part" is the inverse of "part_of", and "born_in" followed by "city_of" gives "nationality". Before learned embeddings, systems mined such rules explicitly. RESCAL (Nickel, Tresp and Kriegel, ICML 2011) factorised the adjacency tensor with a full matrix per relation, $f=h^\top M_r t$.

## Score functions

> Figure: The pipeline. (1) Triples of a small knowledge graph. (2) Each entity and relation indexes a row of an embedding table. (3) A score function turns the three vectors into one number; which one decides the relation patterns the model can express. (4) Training pushes true triples above corrupted ones. (5) Evaluation ranks every entity for a query, removes the other known answers, and reports MRR and Hits@k.

$$
\begin{aligned}\text{TransE: }&f=-\lVert h+r-t\rVert_{1\text{ or }2} &\quad \text{DistMult: }&f=\textstyle\sum_i h_i r_i t_i\\ \text{ComplEx: }&f=\operatorname{Re}\textstyle\sum_i h_i r_i \bar t_i &\quad \text{RotatE: }&f=-\lVert h\circ r-t\rVert,\ \ r_i=e^{i\theta_i}\end{aligned}
$$

- $h,\ r,\ t$: head, relation and tail embeddings; real in TransE and DistMult, complex ($\mathbb{C}^d$) in ComplEx and RotatE
- $\bar t$: complex conjugate; it makes the ComplEx score change when $h$ and $t$ swap
- $\circ$: element-wise product; with $|r_i|=1$ each coordinate of $h$ is rotated by $\theta_i$

**TransE** models a relation as a translation: $h+r\approx t$, the same geometry as the **Word2Vec** analogy $\textit{king}-\textit{man}+\textit{woman}\approx\textit{queen}$. It keeps entity vectors at unit norm. A symmetric relation needs $h+r=t$ and $t+r=h$, so $r=0$ and $h=t$. A one-to-many relation forces all tails onto one point. TransH (Wang et al., AAAI 2014) and TransR (Lin et al., AAAI 2015) project entities into a relation-specific space to ease this.

**DistMult** is RESCAL with a diagonal $M_r$. It has $d$ parameters per relation instead of $d^2$, but its score is symmetric in $h$ and $t$, so it predicts (France, capital_of, Paris) as readily as the true triple. **ComplEx** moves to complex vectors and conjugates the tail. A real $r$ gives a symmetric relation and an imaginary $r$ an antisymmetric one. **RotatE** constrains every $r_i$ to the unit circle. Symmetry is $\theta_i\in\{0,\pi\}$, an inverse relation is the conjugate rotation, and composition adds angles.

Relation patterns each score function can express (Sun et al., 2019, Table 2)

| Model | Symmetry | Antisymmetry | Inversion | Composition |
| --- | --- | --- | --- | --- |
| TransE | ✗ | ✓ | ✓ | ✓ |
| DistMult | ✓ | ✗ | ✗ | ✗ |
| ComplEx | ✓ | ✓ | ✓ | ✗ |
| RotatE | ✓ | ✓ | ✓ | ✓ |

## Training

Only positive triples are given. Negatives are made by corrupting the head or the tail of a true triple with a random entity. TransE minimises the margin ranking loss $\sum[\gamma+d(h,r,t)-d(h',r,t')]_+$. RotatE uses a log-sigmoid loss with **self-adversarial** sampling, which weights each negative by how plausible the current model finds it:

$$
\mathcal{L}=-\log\sigma\big(\gamma-d_r(h,t)\big)-\sum_{i=1}^{n}p(h'_i,r,t'_i)\,\log\sigma\big(d_r(h'_i,t'_i)-\gamma\big),\qquad p(h'_j,r,t'_j)=\frac{\exp\alpha f(h'_j,r,t'_j)}{\sum_i\exp\alpha f(h'_i,r,t'_i)}
$$

- $d_r(h,t)$: the distance $\lVert h\circ r-t\rVert$; the score is $-d_r$
- $\gamma$: a fixed margin
- $\alpha$: sampling temperature; the weights $p$ are treated as constants (no gradient)

Easy negatives such as (Paris, capital_of, Banana) give almost no gradient after a few epochs. Weighting by plausibility spends the update on hard ones like (Paris, capital_of, Belgium).

## Evaluation: filtered ranks

$$
\text{MRR}=\frac{1}{|Q|}\sum_{q\in Q}\frac{1}{\operatorname{rank}_q},\qquad \text{Hits@}k=\frac{1}{|Q|}\sum_{q\in Q}\mathbb{1}[\operatorname{rank}_q\le k]
$$

- $Q$: a tail query $(h,r,?)$ and a head query $(?,r,t)$ for every test triple
- $\operatorname{rank}_q$: position of the true entity among all entities, after removing every other entity that forms a known true triple (train, validation or test): the **filtered** setting of Bordes et al.

Without filtering, a model that ranks Lyon first for (?, located_in, France) is penalised even though Lyon is a correct answer. Filtered MRR is the headline number in most papers, usually with Hits@1, @3 and @10.

Worked example: TransE in 2-D with r = located_in = (1, 0). Entities: Paris (0, 1), Lyon (0.1, 0.8), Madrid (0, −1), Rome (0.3, 0.5), France (1.1, 0.9), Spain (0.9, −1.2), Italy (1.4, −0.2). (Lyon, located_in, France) is a training triple.

| Query | Prediction | Nearest entities (distance) | Raw rank | Filtered rank |
| --- | --- | --- | --- | --- |
| (Paris, located_in, ?) → France | h + r = (1, 1) | France 0.141, Rome 0.860 | 1 | 1 |
| (Madrid, located_in, ?) → Spain | h + r = (1, −1) | Spain 0.224, Italy 0.894 | 1 | 1 |
| (Rome, located_in, ?) → Italy | h + r = (1.3, 0.5) | France 0.447, Italy 0.707 | 2 | 2 |
| (?, located_in, France) → Paris | t − r = (0.1, 0.9) | Lyon 0.100, Paris 0.141 | 2 | 1 (Lyon is known) |
| Metrics over the 4 queries |  |  | MRR 0.75, Hits@1 0.50 | MRR 0.875, Hits@1 0.75 |

**Benchmarks and results**

- **FB15k-237:** Toutanova and Chen (2015): 14,541 entities, 237 relations; 272,115 / 17,535 / 20,466 train / valid / test triples
- **WN18RR:** Dettmers et al. (AAAI 2018): 40,943 WordNet synsets, 11 relations; 86,835 / 3,034 / 3,134 triples
- **Why the new splits:** in the older FB15k and WN18, many test triples are inverses of training triples, so a simple inverse-relation rule scored near the top; the -237 and RR versions remove such relations
- **DistMult / ComplEx:** filtered MRR 0.241 / 0.247 on FB15k-237 and 0.43 / 0.44 on WN18RR, as reported by Dettmers et al. (2018)
- **RotatE:** filtered MRR 0.338, Hits@10 0.533 on FB15k-237; MRR 0.476, Hits@10 0.571 on WN18RR (Sun et al., 2019, with self-adversarial sampling)

> **Training tricks move the numbers as much as the model:** Ruffinelli, Broscheit and Gemulla ("You CAN Teach an Old Dog New Tricks!", ICLR 2020) re-tuned RESCAL, TransE, DistMult and ComplEx with modern losses, negative sampling and regularisation and found them competitive with much later models. Lacroix, Usunier and Obozinski (ICML 2018) showed the same for ComplEx with reciprocal relations and an N3 regulariser. Compare models only under one training protocol.

> Figure: TransE and RotatE trained in the browser (margin 0.5, SGD lr 0.05, 2 negatives per triple, entities on the unit circle) on 9 triples: Ann → Cal → Eve and Ben → Dee → Fay under manages, and peer_of between Ann–Ben and Cal–Dee in both directions and Fay → Eve. The test triple (Eve, peer_of, Fay) is held out. Default (TransE, epoch 200): ‖r_peer_of‖ = 0.065 against ‖r_manages‖ = 0.815, and the partners collapse (Ann 104.4°, Ben 104.5°). Fay at 0.046 beats Eve herself at 0.065, but at 13 of the 19 snapshots from epoch 20 Eve is nearer and the held-out rank is 2 (epoch 190: 0.101 vs 0.160). Train MRR 0.722, Hits@1 44%. Switch to RotatE: loss 0 from epoch 20, θ_peer_of = 182.9°, θ_manages = 239.7°, train MRR 1.000, and the held-out triple ranks first both ways (Fay 0.284, next Ann 0.940).

## Connection to graph neural networks

All four models are shallow: the encoder is a lookup table, as in DeepWalk (see **Graph representation learning**). A GNN can replace it. The **Relational GCN** (Schlichtkrull et al., ESWC 2018) computes entity vectors by message passing with one weight matrix per relation and scores triples with a DistMult decoder; CompGCN (Vashishth et al., ICLR 2020) also embeds the relations and composes them with entities using TransE-, DistMult- or circular-correlation operators. Message passing lets an entity with few facts borrow evidence from its neighbours, at the cost of a far more expensive forward pass.

```python
import numpy as np

# score functions: higher = more plausible
transe   = lambda h, r, t: -np.linalg.norm(h + r - t, axis=-1)
distmult = lambda h, r, t: np.sum(h * r * t, axis=-1)
complex_ = lambda h, r, t: np.real(np.sum(h * r * np.conj(t), axis=-1))
rotate   = lambda h, theta, t: -np.linalg.norm(h * np.exp(1j * theta) - t, axis=-1)

rng = np.random.default_rng(0)
d = 50
h, t = rng.normal(size=(2, d))
hc, tc = rng.normal(size=(2, d)) + 1j * rng.normal(size=(2, d))
r = rng.normal(size=d)

print(np.isclose(distmult(h, r, t), distmult(t, r, h)))                    # True: DistMult is always symmetric
print(np.isclose(complex_(hc, r + 0j, tc), complex_(tc, r + 0j, hc)))      # True: real r -> symmetric
print(np.isclose(complex_(hc, 1j * r, tc), -complex_(tc, 1j * r, hc)))     # True: imaginary r -> antisymmetric
th = np.full(d, np.pi)                                                       # rotation by pi is its own inverse
print(np.isclose(rotate(hc, th, -hc), 0), np.isclose(rotate(-hc, th, hc), 0))   # True True: h -> -h and back

def filtered_rank(scores, truth, known):
    """Rank of the true entity among all candidates, ignoring other known true answers."""
    mask = np.ones_like(scores, dtype=bool)
    mask[list(known - {truth})] = False
    return 1 + int(np.sum(scores[mask] > scores[truth]))

# the worked example: TransE in 2-D, r = located_in = (1, 0)
names = ["Paris", "Lyon", "Madrid", "Rome", "France", "Spain", "Italy"]
E = np.array([[0, 1], [0.1, 0.8], [0, -1], [0.3, 0.5], [1.1, 0.9], [0.9, -1.2], [1.4, -0.2]])
loc = np.array([1.0, 0.0])
ranks = [
    filtered_rank(transe(E[0], loc, E), 4, set()),          # (Paris, located_in, ?)  -> France
    filtered_rank(transe(E[2], loc, E), 5, set()),          # (Madrid, located_in, ?) -> Spain
    filtered_rank(transe(E[3], loc, E), 6, set()),          # (Rome, located_in, ?)   -> Italy
    filtered_rank(transe(E, loc, E[4]), 0, {1}),            # (?, located_in, France) -> Paris; Lyon is known
]
ranks = np.array(ranks)
print(ranks, "MRR", (1 / ranks).mean(), "Hits@1", (ranks <= 1).mean())   # [1 1 2 1] MRR 0.875 Hits@1 0.75
```

The four score functions, their relation patterns, and filtered ranks on the worked example (numpy, runs as is)

- **Knowledge-base completion:** Suggesting missing facts in Freebase-, Wikidata- or enterprise-scale graphs for human review.
- **Biomedical discovery:** Ranking drug–target or drug–disease links in graphs of genes, proteins, compounds and diseases, e.g. for drug repurposing.
- **Recommendation and search:** Item and user embeddings that respect typed relations (bought, belongs_to, made_by) and feed retrieval or ranking models.

**Question:** Why can DistMult not model the relation parent_of correctly?

- A. It has too few parameters per relation
- B. Its score Σ h_i r_i t_i is unchanged when head and tail swap, so (child, parent_of, parent) scores as high as the true triple
- C. It cannot be trained with negative sampling
- D. It requires complex-valued embeddings

**Answer:** B. Its score Σ h_i r_i t_i is unchanged when head and tail swap, so (child, parent_of, parent) scores as high as the true triple — The product $h_i r_i t_i$ is commutative in $h$ and $t$, so every DistMult relation is symmetric. ComplEx breaks the symmetry with the conjugate $\bar t$; RotatE with any rotation other than $0$ or $\pi$.

**Question:** For (?, located_in, France), the true answer Paris is ranked 2nd, behind Lyon, which is also located in France and appears in the training set. What is the filtered rank?

- A. 1
- B. 2
- C. It is undefined
- D. 0.5

**Answer:** A. 1 — Filtering removes every other entity that forms a known true triple with the query, here Lyon. Paris is then first. The raw rank, 2, would contribute $1/2$ to MRR instead of $1$.

**Key points**

- A KG embedding is a lookup table for entities and relations plus a score $f(h,r,t)$; completion ranks all entities for $(h,r,?)$ and $(?,r,t)$.
- TransE: $-\lVert h+r-t\rVert$, translations; cannot express symmetry ($r=0$ forces $h=t$) and struggles with one-to-many relations.
- DistMult: $\sum h_ir_it_i$, cheap but always symmetric. ComplEx: $\operatorname{Re}\sum h_ir_i\bar t_i$, adds antisymmetry and inversion.
- RotatE: $-\lVert h\circ r-t\rVert$ with $|r_i|=1$; rotations give symmetry ($\theta=\pi$), inversion (conjugate) and composition (adding angles).
- Train with corrupted negatives (margin loss, or log-sigmoid with self-adversarial weights); report filtered MRR and Hits@k on FB15k-237 and WN18RR, not the leaky FB15k and WN18.
- Training protocol matters as much as the score function; GNN encoders such as R-GCN can replace the lookup table.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
