Google Research has released Agile Deliberation, a framework for building image classifiers around subjective concepts like ‘healthy food’ or ‘inspiring artwork’. Most tools assume users already know where a concept ends. Agile Deliberation assumes they do not, and helps them find the boundary.
TL;DR
- Framework: It prompts Gemini 2.5 Pro and Gemini 2.5 Flash through the API, without fine-tuning.
- Runs on: A Jupyter or Colab notebook. Retrieval uses CLIP embeddings and a FAISS index on CPU or GPU. The demo index starts at about 260 MB.
- Performance: 10.5 F1 points above zero-shot Gemini 2.5 Flash on average, across 2 concepts and 18 live sessions.
- Best: Paid to play F1 of 0.59 vs 0.48 zero-shot, with precision up 22 points (0.35 to 0.57).
- Worst: Recall drops 17 points on paid to play (0.80 to 0.63) and 15 points on healthy food.
What is Agile Deliberation?
Agile Deliberation turns a vague concept into a structured written definition. That definition serves as a human-readable policy and as the text prompt for a VLM classifier. Each round keeps the candidate definition with the best F1 on the user’s labels.
The design came from interviews with a group of content moderation experts. Experts scoped a concept first, then refined it on borderline images. Finding those images was the slow part.
How does concept scoping work?
Stage 1 breaks the concept into at most 3 unit concepts, using prompt chaining with Gemini 2.5 Pro. Each unit gets candidate positive and negative subconcepts, shown with retrieved images. For healthy food, positives might be salads or grain bowls. Negatives might be processed food or raw, unprepared ingredients. The user keeps, flips or drops each one.
How does concept iteration find borderline images?
Stage 2 skips classic uncertainty sampling. The classifier is a prompted generative VLM, so its confidence scores are often uncalibrated. Instead, the system hunts for images that are borderline in meaning.
The pipeline runs 4 steps:
- Gemini 2.5 Flash writes borderline queries, such as ‘salads with heavy mayo dressings’. Each query retrieves 50 to 100 images.
- Images are de-duplicated, encoded, and grouped into clusters with dictionary learning.
- A UCB multi-armed bandit picks the next cluster. It favors clusters where users corrected the model or left rich feedback.
- A VLM summarizes each image in 1 sentence. The user sees at least 5 images that share 1 ambiguity.
Users label each image and add short notes on disagreements. An LLM turns those notes into candidate definitions. A greedy update keeps the best-scoring one.
How was it evaluated?
Subjective concepts have no fixed ground truth, so Google team skipped static benchmarks. It ran 18 live sessions of about 90 minutes each. 9 participants without professional moderation experience did 2 sessions each. One session used Agile Deliberation, the other used manual deliberation. Each participant spent about 45 minutes defining a concept, then labeled 200 held-out test images.
The two concepts were ‘paid to play’ clickbait and ‘healthy food’. Retrieval ran over a proprietary 100M-image dataset and WebLI, respectively.
What do the benchmarks show?
| F1 (Agile Deliberation group) | Zero-shot | Modeling Collaborator | Agile Deliberation | Gain vs zero-shot |
|---|---|---|---|---|
| Paid to play | 0.48 | 0.53 | 0.59 | +11 points |
| Healthy food | 0.48 | 0.50 | 0.58 | +10 points |
Source: Table 1 of the CVPR 2026 paper. F1 averaged over participants.
How does it compare with other subjective classifier methods?
| Feature | Agile Deliberation | Modeling Collaborator | Agile Modeling | Zero-shot VLM |
|---|---|---|---|---|
| Venue | CVPR 2026 | CVPR 2024 | ICCV 2023 | Baseline in paper |
| Core idea | Deliberation on semantic borderline images, then automatic prompt optimization | LLM agent decomposes the concept and auto-labels data | User labels images in a real-time loop | Classify from the initial description |
| Human effort | About 45 min of labeling and feedback | About 100 labels plus chat (vs about 2,000) | Under 30 min in a 14-person study | 1 concept description |
| Handles evolving definitions | Yes | No (static) | No (static) | No |
| Classifier | Prompted Gemini 2.5 Flash | Lightweight distilled classifier | Not disclosed | Prompted Gemini 2.5 Flash |
| F1, paid to play | 0.59 | 0.53 | Not tested | 0.48 |
| F1, healthy food | 0.58 | 0.50 | Not tested | 0.48 |
| Code | Apache 2.0, GitHub | Not disclosed | Not disclosed | Gemini API |
How can developers run it?
The repo ships a Demo.ipynb that walks through the healthy food example. It needs a Gemini API key and defaults to gemini-2.5-pro and gemini-2.5-flash. A setup script builds a FAISS index over LAION-400M CLIP embeddings. The default 2 shards take about 260 MB. All 100 shards take about 13 GB.
What it means
For people building classifiers, the change is practical. You do not need to guess the definition of a subjective term like ‘healthy food’ before starting. The system asks you to define it as you go. It presents borderline images to help you refine the rule. This reduces the time spent on prompt editing and improves the final accuracy.
