Embedding models

A bi-encoder maps each text to one unit vector so relevance is a dot product: trained contrastively with in-batch negatives, chosen on MTEB-style task scores, and sized by dimensions and precision; Matryoshka training makes prefixes usable.

8 min read739 wordscore0 viewsupdated 2026-10-21

Definition

Embedding model an encoder ff that maps a text (or image) to a fixed-length vector f(x)∈Rdf(x)\in\mathbb{R}^d such that related inputs get a high similarity, usually the cosine. In retrieval it is used as a bi-encoder: queries and documents are embedded separately, so documents are embedded once, offline. The design comes from Sentence-BERT (Reimers and Gurevych, 2019) and DPR (Karpukhin et al., 2020). MTEB (Muennighoff et al., 2022) is the standard benchmark for comparing them.

In a RAG system the embedding model decides what "relevant" means. It embeds every chunk at ingestion and every query at request time, and the vector index (see Vector databases) returns the chunks whose vectors are closest. Its errors cannot be fixed downstream except by a reranker that sees fewer candidates. How pooling and normalisation work inside the encoder is covered in Sentence embeddings; this article is about choosing, training and deploying the model.

A cross-encoder reads query and document together and is more accurate, but it must run once per pair. With 10710^7 chunks that is 10710^7 forward passes per query. A bi-encoder needs one forward pass per query plus a nearest-neighbour search, which is why it does first-stage retrieval and the cross-encoder only reranks (see Reranking).

inference · bi-encoder: each text is embedded alone, passages once and offlinequery: how do I resetmy router?passage: Hold the reset buttonfor 10 s to restore defaults.encoder(sameweights)12 layersmean-pooltokens↓÷ ‖v‖0.31-0.120.050.44… ×7680.28-0.090.110.39… ×768score = q · d = cos(q, d)unit length: ‖q‖ = ‖d‖ = 1training · in-batch contrastive (B = 4 shown)p1p2p3p4q10.820.210.080.15q20.120.770.300.05q30.180.260.710.11q40.040.090.140.86row-wise softmax over cos/τ:q_i must pick p_i (diagonal)off-diagonal = free negatives;add mined hard negatives(BM25 hits that are wrong)Matryoshka loss64128256512768 dimsL = Σ_m L_contrastive(v[:m])storage per 1M vectors (raw vectors only; the ANN index adds more)1024-d float324096 B / vector4.10 GB256-d float321024 B / vector1.02 GB256-d int8256 B / vector0.26 GB1024-d binary128 B / vector0.13 GB
fig.A bi-encoder at inference and in training. Query and passage go through the same encoder (with E5-style prefixes), are mean-pooled and normalised to unit length, and are compared by a dot product. Training fills a batch similarity matrix and pushes the diagonal up. Matryoshka training adds the same loss on nested prefixes of the vector. The bottom row shows what dimensions and precision cost per million vectors.

Contrastive training

L=−1B∑i=1Blog⁡exp⁡ ⁣(s(qi,pi)/τ)∑j=1Bexp⁡ ⁣(s(qi,pj)/τ)+∑hexp⁡ ⁣(s(qi,ph−)/τ),s(q,p)=f(q)⊤f(p)∥f(q)∥ ∥f(p)∥\mathcal{L}=-\frac{1}{B}\sum_{i=1}^{B}\log\frac{\exp\!\big(s(q_i,p_i)/\tau\big)}{\sum_{j=1}^{B}\exp\!\big(s(q_i,p_j)/\tau\big)+\sum_{h}\exp\!\big(s(q_i,p_h^-)/\tau\big)},\qquad s(q,p)=\frac{f(q)^\top f(p)}{\lVert f(q)\rVert\,\lVert f(p)\rVert}
(qi,pi)(q_i,p_i)
a positive pair: question and answer passage, title and body, query and clicked result
pj, j≠ip_j,\ j\neq i
the other passages in the batch, used as free negatives (in-batch negatives)
ph−p_h^-
mined hard negatives: passages that look relevant (e.g. high BM25 score) but are not
τ\tau
temperature; small values (around 0.01–0.05) sharpen the softmax

This is InfoNCE (van den Oord et al., 2018) applied to text pairs; the derivation and its batch-size effects are in Contrastive language learning. Modern general-purpose models (E5, BGE, GTE and others) train in two stages: weakly supervised pre-training on hundreds of millions of web pairs, then fine-tuning on labelled retrieval and similarity data with hard negatives.

Many retrieval models are asymmetric: short queries and long passages get different input formats. E5 (Wang et al., 2022) expects the prefixes query: and passage: ; BGE English models add an instruction to the query only. Omitting the prefix the model was trained with still returns vectors, just worse ones.

Dimensions, precision and Matryoshka embeddings

LMRL=∑m∈Mcm L(v^1:m),v^1:m=v1:m∥v1:m∥,bytes=N⋅d⋅b8\mathcal{L}_{\text{MRL}}=\sum_{m\in\mathcal{M}}c_m\,\mathcal{L}\big(\hat v_{1:m}\big),\qquad \hat v_{1:m}=\frac{v_{1:m}}{\lVert v_{1:m}\rVert},\qquad \text{bytes}=N\cdot d\cdot\frac{b}{8}
M\mathcal{M}
nested prefix sizes, e.g. {64,128,256,512,768}\{64,128,256,512,768\}; cmc_m are weights (often all 1)
v1:mv_{1:m}
the first m coordinates of the embedding, re-normalised before use
N, d, bN,\,d,\,b
number of vectors, dimensions kept, bits per dimension (32 for float32, 8 for int8, 1 for binary)

Matryoshka Representation Learning (Kusupati et al., NeurIPS 2022) trains with the loss on every prefix in M\mathcal{M}, so the important information is packed into the first coordinates. You can then store a 256-d prefix of a 768-d vector, or search with a short prefix and rescore the shortlist with the full vector. Truncating a model trained without this loss throws away information spread evenly over all coordinates.

Precision is the other lever. Scalar int8 quantisation stores one byte per dimension with a per-dimension scale and usually loses little recall. Binary quantisation keeps only the sign bit (32× smaller than float32) and is normally paired with rescoring of a shortlist using the full-precision vectors.

Reference points

MTEB
Muennighoff et al. (arXiv 2022, EACL 2023): 8 task types (bitext mining, classification, clustering, pair classification, reranking, retrieval, STS, summarisation), 58 datasets, 112 languages; no single model was best on every task type
Matryoshka
Kusupati et al. (NeurIPS 2022): reported up to 14× smaller embeddings at the same ImageNet-1K classification accuracy, and nested prefixes that each work as a standalone embedding
OpenAI v3
text-embedding-3-small (1536-d) and -large (3072-d), January 2024, accept a dimensions parameter; OpenAI reported that -large cut to 256 dims still beat the 1536-d text-embedding-ada-002 on MTEB
Common sizes
all-MiniLM-L6-v2: 384-d, inputs truncated at 256 word pieces; BERT-base-sized encoders: 768-d; BERT-large-sized: 1024-d
Normalised vectors
for unit vectors ∥u−v∥2=2−2cos⁡(u,v)\lVert u-v\rVert^2=2-2\cos(u,v), so cosine, dot-product and L2 search return the same ranking
600 docs, 80 queries, 128-d toy embeddings · cosine (unit vectors) · float32recall@10 vs dims kept (prefix + re-normalise)0.000.250.500.751.00163264128dimensions mMatryoshka (info in early dims)standard (info spread evenly)Matryoshka, m = 32recall@10 = 0.938(75 of 80 queries find their document)128 B per vector (25.0% of 128-d float32)10M vectors: 1.28 GB (before ANN overhead)rank of the correct documentrank 1572–101811–1005> 1000top-1 result is a long document in 21% of queries; long documents are 14.5% of the corpusunit vectors: cosine = dot product, so document length does not change the scoreat m = 128 both models are the same space up to a rotation, so their recall is identical
model
storagescore
fig.A toy benchmark: 600 documents, 80 queries, 128-d embeddings, real nearest-neighbour search. Default (Matryoshka, 32 dims, float32, cosine): 75 of 80 queries find their document in the top 10, at a quarter of the full storage. Truncated to 16 dims, the Matryoshka model still scores 67/80 while the standard model falls to 19/80 (46/80 at 32 dims). At 128 dims both reach 80/80. Switch to raw dot product: long documents win, and recall at 128 dims drops to 51/80 as the top hit is a long document in 86% of queries (they are 14.5% of the corpus). With cosine, int8 storage changes recall by at most 2 of 80 queries.

Worked example

QueryPositive cosBest negative cosp(positive), τ = 1Loss, τ = 1p(positive), τ = 0.1Loss, τ = 0.1
q10.820.210.39490.92910.99590.0041
q20.770.300.37970.96840.98880.0113
q30.710.260.36041.02070.98180.0184
q40.860.140.41840.87140.99850.0015
mean0.94740.0088
InfoNCE on the diagram's 4 × 4 batch (no hard negatives): probability of the right passage and loss per query

With τ=1\tau=1, cosines in [−1,1][-1,1] barely separate in the softmax: a perfectly ordered batch still has mean loss 0.9474, not far below the ln⁡4=1.3863\ln 4=1.3863 of guessing. With τ=0.1\tau=0.1 the same scores give almost all mass to the diagonal; the gradient then concentrates on the hardest row, q3, whose positive (0.71) is closest to a negative.

Storage for 10M chunks

  1. 11024-d float32: 107×1024×410^7\times1024\times4 B =40.96=40.96 GB of raw vectors, before any ANN graph or metadata.
  2. 2Matryoshka prefix of 256 dims, int8: 107×256×110^7\times256\times1 B =2.56=2.56 GB, a factor of 40.96/2.56=1640.96/2.56=16.
  3. 3The query still costs one encoder pass; search over the smaller vectors is cheaper in memory bandwidth, and rescoring the top 100 with the float32 vectors can recover much of the lost precision.
python
1import numpy as np
2from sentence_transformers import SentenceTransformer
3
4model = SentenceTransformer("intfloat/e5-base-v2") # 768-d, expects prefixes
5docs = ["Hold the reset button for 10 s to restore defaults.", "..."]
6D = model.encode([f"passage: {d}" for d in docs], normalize_embeddings=True)
7q = model.encode("query: how do I reset my router?", normalize_embeddings=True)
8
9def prefix(X, m): # only meaningful for Matryoshka-trained models
10 X = X[..., :m]
11 return X / np.linalg.norm(X, axis=-1, keepdims=True)
12
13D256, q256 = prefix(D, 256), prefix(q, 256)
14scale = np.abs(D256).max(axis=0) / 127 # per-dimension int8 scale
15D8 = np.clip(np.round(D256 / scale), -127, 127).astype(np.int8)
16
17coarse = (D8 * scale) @ q256 # cheap first pass
18top = np.argsort(-coarse)[:100]
19final = top[np.argsort(-(D[top] @ q))] # rescore with full vectors
Embedding with prefixes, a Matryoshka-style prefix and int8 storage (sentence-transformers; the model download could not be run here)

Use when

  • +One encoder pass per query; documents are embedded once, so search over millions of chunks is fast
  • +Matches paraphrases and cross-lingual queries that share no words with the document
  • +Dimensions and precision can be traded for storage, especially with Matryoshka training

Avoid when

  • −Less precise than a cross-encoder; negation, numbers and exact identifiers are weak spots (pair with BM25, see Hybrid search (BM25 + dense))
  • −Inputs beyond the maximum sequence length are silently truncated
  • −Every model change forces a full re-embedding of the corpus

Check yourself

You truncate the 768-d vectors of a model trained without a Matryoshka loss to 128 dims. What should you expect?

Check yourself

Your vectors are not normalised and you search with the dot product. Which documents are favoured?

Remember

  • ✓Bi-encoders embed queries and documents separately; cross-encoders are more accurate but only affordable for reranking.
  • ✓Training: InfoNCE over in-batch negatives plus mined hard negatives, temperature τ\tau around 0.01–0.05.
  • ✓Use the prefixes or instructions the model was trained with, and normalise if it was trained on cosine.
  • ✓Storage =N⋅d⋅b/8=N\cdot d\cdot b/8: 10M × 1024-d float32 is 40.96 GB; a 256-d int8 Matryoshka prefix is 2.56 GB.
  • ✓Choose on MTEB, then confirm on your own labelled queries; store the model version with every vector.
← back to Generative AI roadmap