Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers

Sentence Transformers v6.0 now supports multi-vector models that keep one vector per token instead of compressing a whole text into a single summary.

Standard embedding models squeeze an entire document into one fixed-size vector. A multi-vector model preserves one vector for every word or token. It scores a query against a document using the MaxSim operator. This approach retains token-level matching details that a single vector usually averages out, typically delivering stronger retrieval results at the cost of a larger index. It is currently the leading method for visual document retrieval, where a text query matches against page images without an OCR step.

This guide covers loading checkpoints, encoding and scoring, integrating them into a search stack, processing page images, and managing index size. All examples run with a standard pip install -U sentence-transformers command.

Table of Contents

• What are Multi-Vector Models?
• Installation
• Loading a Model
• Encoding Queries and Documents
• Scoring with MaxSim
• Semantic Search
• Retrieve and Rerank
• Indexing
• Visual Document Retrieval
• Audio Retrieval
• Video Retrieval
• Interpretability
• Token Pooling
• Speeding Up Inference
• Evaluating a Model
• Coming from PyLate or colpali-engine
• Supported Models
• Acknowledgements
• Additional Resources

What are Multi-Vector Models?

A dense embedding model reads text and returns a single fixed-size vector. Everything the model notices must fit into 384, 768, or 1024 numbers. Similarity is calculated as one dot product between two such summaries. This works remarkably well, but the compression is lossy. A rare entity, an exact identifier, or one crucial clause in a long passage must compete for space in the same vector. A query with several requirements at once hits the same limit.

For a search like “green sofa with wooden legs and rounded cushions”, a single vector blends all four elements into one point. A green sofa with the wrong legs ends up close to the one you actually asked for.

A multi-vector model skips that compression. It runs the same transformer but projects each token embedding down to a small dimension, classically 128, and keeps all of them. A nine-token document becomes a 9×128 matrix, not a 1×128 vector.

The interaction between query and document is deferred until scoring time. This is where the name “late interaction” comes from. A cross-encoder interacts early by feeding both texts through the model together. It is accurate but leaves nothing to precompute, since every document must be re-encoded for each new query. A bi-encoder, which is what the dense embedding model above is, barely interacts at all. That is exactly what lets you encode a collection once and query it fast. Late interaction sits in between: documents are still encoded independently and can be indexed offline, but scoring compares every query token against every document token, leaving far more room for the two to interact.

The MaxSim Operator

Scoring uses MaxSim. For each query token, take its highest similarity against any document token, then sum those maxima across the query.

Because the token embeddings are L2-normalized, each of those dot products is a cosine similarity in [-1, 1]. The whole sum lands within [-num_query_tokens, num_query_tokens].

You can read the operator as a soft alignment. Every query token points at the one document token that best explains it, and the score is how well the document supports the query overall.

The alignment does not have to be lexical, since the token embeddings are contextualized. Encode “Where do penguins live?” against “Penguins inhabit Antarctica.” with lightonai/mLateOn and the query token live finds its best match on inhabit at 0.94. It is a word it shares no characters with. That is the thing lexical retrieval cannot do. BM25 and its relatives need the term itself, so synonyms and paraphrases slip past them. Dense embedding models bridge that gap as well, of course. What late interaction adds is that it does so without giving up the other direction: when an exact match is what matters, a product code, a surname, or a function name, MaxSim still has that token sitting there on its own, where a single-vector model had to average it in with everything else. It is not one-to-one either, since several query tokens routinely settle on the same document token.

What You Gain, and What It Costs

You gain retrieval quality, particularly on queries where one specific piece of a document is what makes it relevant, on multi-requirement queries like the sofa above where each requirement gets to find its own evidence, and on out-of-domain data where a dense model’s compression was tuned for a different distribution. That compression is learned from the training queries, so the model learns to keep what they needed and drop everything else, which may include exactly what your production queries ask about. The effect grows with document length, since more text has to fit in the same fixed vector.

The cost is index size. One vector per token instead of one vector per document is a lot more vectors, only partly offset by the smaller dimension. Encoding 4,874 Natural Questions passages with lightonai/LateOn produced 608,414 token vectors, an average of 124.8 per passage:

That is about 42 times the storage of the MiniLM index, or 62 KiB per passage. However, indexes are often compressed. The same 608,414 vectors take 92 MB as a fast-plaid index, since PLAID stores a centroid id plus a quantized residual per vector rather than the vector itself. For scale, a 4096-dimensional dense model like Qwen3-Embedding-8B would need about 80 MB for these same 4,874 passages, so a compressed multi-vector index sits in the same territory as the dense indexes people already run. Token Pooling cuts the vector count before any of that, and Retrieve and Rerank avoids building an index at all.

PyLate comes up throughout this post, so briefly: Sentence Transformers handled dense and sparse models but not late interaction, so LightOn built PyLate on top of it to close that gap, adding the training, inference, and retrieval pieces these models need. Much of what you will load below was trained with it, and LightOn built an ecosystem around it too, including fast-plaid, the late-interaction index that turns up in Indexing. With v6.0 those capabilities live in Sentence Transformers itself.

With the tradeoff in mind, let’s get a model running.

Installation

Multi-vector models work with a plain install:

pip install -U sentence-transformers

For ColPali-style visual document retrieval, you also need the image dependencies (see Installation for all extras, and Multimodal Embedding & Reranker Models for multimodal support in general):

pip install -U "sentence-transformers[image]"

Sentence Transformers v6.0 requires transformers v5.x, torch 2.2+, and huggingface-hub v1.x. If you pin any of those lower, plan the upgrade first. See the Migration Guide for the full list of breaking changes.

Loading a Model

Loading a multi-vector model looks exactly like loading any other Sentence Transformers model:

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("lightonai/LateOn")

To find models that work, look for the multi-vector and sentence-transformers tags on the Hub. Any model with those tags loads with the line above, whether it started life as a PyLate checkpoint, a Stanford-NLP ColBERT checkpoint, or a ColPali-family model for visual document retrieval. We are working through the ecosystem to get that tag onto every model that works, so the list keeps growing.

Underneath, MultiVectorEncoder reads each of the formats these checkpoints have been published in over the years. PyLate and Stanford-NLP checkpoints load directly even where the tag has not been added yet:

from sentence_transformers import MultiVectorEncoder

# Native Sentence Transformers checkpoints. PyLate builds on the same schema,
# so any PyLate checkpoint loads identically
model = MultiVectorEncoder("lightonai/LateOn")
model = MultiVectorEncoder("mixedbread-ai/mxbai-edge-colbert-v0-17m")
model = MultiVectorEncoder("LiquidAI/LFM2.5-ColBERT-350M", trust_remote_code=True)

# Any Stanford-NLP ColBERT checkpoint, detected via the `HF_ColBERT` architecture
# marker. The inline projection weight and the recipe come from `artifact.metadata`
model = MultiVectorEncoder("colbert-ir/colbertv2.0")
model = MultiVectorEncoder("answerdotai/answerai-colbert-small-v1")

# A bare transformer: a fresh random projection is appended, so training is required
model = MultiVectorEncoder("answerdotai/ModernBERT-base")

Visual document retrieval models are the exception. ColPali-family checkpoints ship in colpali-engine’s own format, which carries no information Sentence Transformers can use, so each one needs a small configuration added to its repository before it loads. Most of that work is done and