“`html

Multimodal Embedding & Reranker Models with Sentence Transformers

Traditional embedding models convert text into fixed-size vectors. Multimodal embedding models extend this by mapping inputs from different modalities (text, images, audio, or video) into a shared embedding space. This means you can compare a text query against image documents (or vice versa) using the same similarity functions you’re already familiar with.

If you want to train your own multimodal models, check out the companion blogpost: Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers.

Table of Contents

• What are Multimodal Models?
• Installation
• Multimodal Embedding Models
• Multimodal Reranker Models
• Retrieve and Rerank
• Input Formats and Configuration
• Supported Models
• Additional Resources

What are Multimodal Models?

Multimodal embedding models map inputs from different modalities into a shared embedding space, while multimodal reranker models score the relevance of mixed-modality pairs. This opens up use cases like visual document retrieval, cross-modal search, and multimodal RAG pipelines.

For example, you can compare a text query against image documents, find video clips matching a description, or build RAG pipelines that work across modalities.

Installation

Multimodal models require some extra dependencies. Install the extras for the modalities you need (see Installation for more details):

# For image support
pip install -U "sentence-transformers[image]

# For audio support
pip install -U "sentence-transformers

# For video support
pip install -U "sentence-transformers"

# Mix and match as needed
pip install -U "sentence-transformers[image,video]

VLM-based models like Qwen3-VL-2B require a GPU with at least ~8 GB of VRAM. For the 8B variants, expect ~20 GB. If you don’t have a local GPU, consider using a cloud GPU service or Google Colab. On CPU, these models will be extremely slow; text-only or CLIP models are better suited for CPU inference.

Multimodal Embedding Models

Loading a Model

Loading a multimodal embedding model works exactly like loading a text-only model:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B")

Some models might require a revision argument for now if the integration pull requests for the model is still pending. Once they’re merged, you’ll be able to load them without specifying a revision, like above.

The model automatically detects which modalities it supports, so there’s nothing extra to configure. See Processor and Model kwargs if you want to control things like image resolution or model precision.

Encoding Images

With a multimodal model loaded, the model.encode() function accepts images alongside text. Images can be provided as URLs, local file paths, or PIL Image objects (see Supported Input Types for all accepted formats):

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B")

# Encode images from URLs
img_embeddings = model.encode([
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
])
print(img_embeddings.shape)
# (2, 2048)

Cross-Modal Similarity

You can compute similarities between text embeddings and image embeddings, since the model maps both into the same space:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B")

# Encode images
img_embeddings = model.encode([
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
])

# Encode text queries (one matching + one hard negative per image)
text_embeddings = model.encode([
    "A green car parked in front of a yellow building",
    "A red car driving on a highway",
    "A bee on a pink flower",
    "A wasp on a wooden table",
])

# Compute cross-modal similarities
similarities = model.similarity(text_embeddings, img_embeddings)
print(similarities)
# tensor([[0.5115, 0.1078],
#         [0.1999, 0.1108],
#         [0.1255, 0.6749],
#         [0.1283, 0.2704]])

As expected, “A green car parked in front of a yellow building” is most similar to the car image (0.51), and “A bee on a pink flower” is most similar to the bee image (0.67). The hard negatives (“A red car driving on a highway”, “A wasp on a wooden table”) correctly receive lower scores.

You might notice that even the best matching scores (0.51, 0.67) aren’t very close to 1.0. This is due to the modality gap: embeddings from different modalities tend to cluster in separate regions of the space. Cross-modal similarities are typically lower than within-modal ones (e.g., text-to-text), but the relative ordering is preserved, so retrieval still works well.

Encoding Queries and Documents

For retrieval tasks, the encode_query() and encode_document() methods are recommended. Many retrieval models prepend different instruction prompts depending on whether the input is a query or a document, similar to how chat models might apply different system prompts depending on the goal. Model authors can specify their prompts in the model config, and encode_query()/encode_document() automatically load and apply the correct one:

• The encode_query() method uses the model’s query prompt (if available) and sets task=”query”.
• The encode_document() method uses the first available prompt from document, passage, or corpus, and sets task=”document”.

Under the hood, both are thin wrappers around encode(), they just handle prompt selection for you. Here’s what cross-modal retrieval looks like:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B")

# Encode text queries with the query prompt
query_embeddings = model.encode_query([
    "Find me a photo of a vehicle parked near a building",
    "Show me an image of a pollinating insect",
])

# Encode document screenshots with the document prompt
doc_embeddings = model.encode_document([
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
])

# Compute similarities
similarities = model.similarity(query_embeddings, doc_embeddings)
print(similarities)
# tensor([[0.3907, 0.1490],
#         [0.1235, 0.4872]])

These methods accept the same input types as encode() (images, URLs, multimodal dicts, etc.) and pass through the same parameters. For models without specialized query/document prompts, they behave identically to encode().

Multimodal Reranker Models

Ranking Mixed-Modality Documents

The rank() method scores and ranks a list of documents against a query, supporting mixed modalities:

from sentence_transformers import CrossEncoder

model = CrossEncoder("Qwen/Qwen3-VL-Reranker-2B")

query = "A green car parked in front of a yellow building"
documents = [
    # Image documents (URL or local file path)
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
    # Text document
    "A vintage Volkswagen Beetle painted in bright green sits in a driveway.",
    # Combined text + image document
    {
        "text": "A car in a European city",
        "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
    },
]

rankings = model.rank(query, documents)
for rank in rankings:
    print(f"{rank['score']:.4f}\t(document {rank['corpus_id']})")
"""
0.9375  (document 0)
0.5000  (document 3)
-1.2500 (document 2)
-2.4375 (document 1)
"""

The reranker correctly identifies the car image (document 0) as the most relevant result, followed by the combined text+image document about a car in a European city (document 3). The bee image (document 1) scores lowest. Keep in mind that the modality gap can influence absolute scores: text-image pair scores may occupy a different range than text-text or image-image pair scores.

You can also check which modalities a reranker supports using modalities and supports(), just like with embedding models:

print(model.modalities)
# ['text', 'image', 'video', 'message]

print(model.supports("image"))
# True

# Check if the model supports a specific pair of modalities
print(model.supports(("image", "text")))
# True

Predicting Pair Scores

You can also use predict() to get raw relevance scores for specific pairs of inputs:

from sentence_transformers import CrossEncoder
model
Source Read original →
Related reading
Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers
Liquid AI Introduces LFM2.5-Embedding-350M and LFM2.5-ColBERT-350M: Dense Bi-Encoder and Late-Interaction Models for Fast Multilingual Search Across 11 Languages
Mira Murati’s Thinking Machines Lab Introduces Interaction Models: A Native Multimodal Architecture for Real-Time Human-AI Collaboration