Definition
Embedding model an encoder that maps a text (or image) to a fixed-length vector such that related inputs get a high similarity, usually the cosine. In retrieval it is used as a bi-encoder: queries and documents are embedded separately, so documents are embedded once, offline. The design comes from Sentence-BERT (Reimers and Gurevych, 2019) and DPR (Karpukhin et al., 2020). MTEB (Muennighoff et al., 2022) is the standard benchmark for comparing them.
In a RAG system the embedding model decides what "relevant" means. It embeds every chunk at ingestion and every query at request time, and the vector index (see Vector databases) returns the chunks whose vectors are closest. Its errors cannot be fixed downstream except by a reranker that sees fewer candidates. How pooling and normalisation work inside the encoder is covered in Sentence embeddings; this article is about choosing, training and deploying the model.
A cross-encoder reads query and document together and is more accurate, but it must run once per pair. With chunks that is forward passes per query. A bi-encoder needs one forward pass per query plus a nearest-neighbour search, which is why it does first-stage retrieval and the cross-encoder only reranks (see Reranking).
Contrastive training
- a positive pair: question and answer passage, title and body, query and clicked result
- the other passages in the batch, used as free negatives (in-batch negatives)
- mined hard negatives: passages that look relevant (e.g. high BM25 score) but are not
- temperature; small values (around 0.01–0.05) sharpen the softmax
This is InfoNCE (van den Oord et al., 2018) applied to text pairs; the derivation and its batch-size effects are in Contrastive language learning. Modern general-purpose models (E5, BGE, GTE and others) train in two stages: weakly supervised pre-training on hundreds of millions of web pairs, then fine-tuning on labelled retrieval and similarity data with hard negatives.
Many retrieval models are asymmetric: short queries and long passages get different input formats. E5 (Wang et al., 2022) expects the prefixes query: and passage: ; BGE English models add an instruction to the query only. Omitting the prefix the model was trained with still returns vectors, just worse ones.
Dimensions, precision and Matryoshka embeddings
- nested prefix sizes, e.g. ; are weights (often all 1)
- the first m coordinates of the embedding, re-normalised before use
- number of vectors, dimensions kept, bits per dimension (32 for float32, 8 for int8, 1 for binary)
Matryoshka Representation Learning (Kusupati et al., NeurIPS 2022) trains with the loss on every prefix in , so the important information is packed into the first coordinates. You can then store a 256-d prefix of a 768-d vector, or search with a short prefix and rescore the shortlist with the full vector. Truncating a model trained without this loss throws away information spread evenly over all coordinates.
Precision is the other lever. Scalar int8 quantisation stores one byte per dimension with a per-dimension scale and usually loses little recall. Binary quantisation keeps only the sign bit (32× smaller than float32) and is normally paired with rescoring of a shortlist using the full-precision vectors.
Reference points
- MTEB
- Muennighoff et al. (arXiv 2022, EACL 2023): 8 task types (bitext mining, classification, clustering, pair classification, reranking, retrieval, STS, summarisation), 58 datasets, 112 languages; no single model was best on every task type
- Matryoshka
- Kusupati et al. (NeurIPS 2022): reported up to 14× smaller embeddings at the same ImageNet-1K classification accuracy, and nested prefixes that each work as a standalone embedding
- OpenAI v3
text-embedding-3-small(1536-d) and-large(3072-d), January 2024, accept adimensionsparameter; OpenAI reported that-largecut to 256 dims still beat the 1536-dtext-embedding-ada-002on MTEB- Common sizes
all-MiniLM-L6-v2: 384-d, inputs truncated at 256 word pieces; BERT-base-sized encoders: 768-d; BERT-large-sized: 1024-d- Normalised vectors
- for unit vectors , so cosine, dot-product and L2 search return the same ranking
Worked example
| Query | Positive cos | Best negative cos | p(positive), τ = 1 | Loss, τ = 1 | p(positive), τ = 0.1 | Loss, τ = 0.1 |
|---|---|---|---|---|---|---|
| q1 | 0.82 | 0.21 | 0.3949 | 0.9291 | 0.9959 | 0.0041 |
| q2 | 0.77 | 0.30 | 0.3797 | 0.9684 | 0.9888 | 0.0113 |
| q3 | 0.71 | 0.26 | 0.3604 | 1.0207 | 0.9818 | 0.0184 |
| q4 | 0.86 | 0.14 | 0.4184 | 0.8714 | 0.9985 | 0.0015 |
| mean | 0.9474 | 0.0088 |
With , cosines in barely separate in the softmax: a perfectly ordered batch still has mean loss 0.9474, not far below the of guessing. With the same scores give almost all mass to the diagonal; the gradient then concentrates on the hardest row, q3, whose positive (0.71) is closest to a negative.
Storage for 10M chunks
- 11024-d float32: B GB of raw vectors, before any ANN graph or metadata.
- 2Matryoshka prefix of 256 dims, int8: B GB, a factor of .
- 3The query still costs one encoder pass; search over the smaller vectors is cheaper in memory bandwidth, and rescoring the top 100 with the float32 vectors can recover much of the lost precision.
1import numpy as np2from sentence_transformers import SentenceTransformer34model = SentenceTransformer("intfloat/e5-base-v2") # 768-d, expects prefixes5docs = ["Hold the reset button for 10 s to restore defaults.", "..."]6D = model.encode([f"passage: {d}" for d in docs], normalize_embeddings=True)7q = model.encode("query: how do I reset my router?", normalize_embeddings=True)89def prefix(X, m): # only meaningful for Matryoshka-trained models10 X = X[..., :m]11 return X / np.linalg.norm(X, axis=-1, keepdims=True)1213D256, q256 = prefix(D, 256), prefix(q, 256)14scale = np.abs(D256).max(axis=0) / 127 # per-dimension int8 scale15D8 = np.clip(np.round(D256 / scale), -127, 127).astype(np.int8)1617coarse = (D8 * scale) @ q256 # cheap first pass18top = np.argsort(-coarse)[:100]19final = top[np.argsort(-(D[top] @ q))] # rescore with full vectorsUse when
- +One encoder pass per query; documents are embedded once, so search over millions of chunks is fast
- +Matches paraphrases and cross-lingual queries that share no words with the document
- +Dimensions and precision can be traded for storage, especially with Matryoshka training
Avoid when
- −Less precise than a cross-encoder; negation, numbers and exact identifiers are weak spots (pair with BM25, see Hybrid search (BM25 + dense))
- −Inputs beyond the maximum sequence length are silently truncated
- −Every model change forces a full re-embedding of the corpus
Check yourself
You truncate the 768-d vectors of a model trained without a Matryoshka loss to 128 dims. What should you expect?
Check yourself
Your vectors are not normalised and you search with the dot product. Which documents are favoured?
Remember
- ✓Bi-encoders embed queries and documents separately; cross-encoders are more accurate but only affordable for reranking.
- ✓Training: InfoNCE over in-batch negatives plus mined hard negatives, temperature around 0.01–0.05.
- ✓Use the prefixes or instructions the model was trained with, and normalise if it was trained on cosine.
- ✓Storage : 10M × 1024-d float32 is 40.96 GB; a 256-d int8 Matryoshka prefix is 2.56 GB.
- ✓Choose on MTEB, then confirm on your own labelled queries; store the model version with every vector.