IBM Research tops the τ²-bench customer service leaderboard with a basic model and learned memory
Learned memory takes a basic ReAct agent with GPT-4.1 to the top
A customer-service agent using GPT-4.1 reaches 67.5% Pass^4 on the τ²-bench retail benchmark. It scores above every retail entry in the October 12, 2026 leaderboard snapshot. The system runs on the benchmark’s stock ReAct scaffold, with learned guidelines appended to the system prompt.
ALTK-Evolve builds that playbook using GPT-4.1. It analyzes the agent’s training-task attempts and uses the model to distill successes and failures into short behavioral rules: how to authenticate a customer, confirm a transaction, or handle a return. Those guidelines are then added to the agent’s prompt for subsequent conversations. The same model powers both guideline extraction and the agent; no weight updates or human annotation are needed.
Our previous work demonstrated this approach on coding agents. Here, we test it on customer service, where success requires following domain policies, understanding a customer’s request, and executing the right tool actions.
τ²-bench tests policy-bound customer service
τ²-bench (from Sierra) measures how well an agent handles customer-service conversations under real constraints. Each task pairs the agent with an LLM-simulated customer who has a goal in mind, a set of domain tools (look up an order, issue a refund, rebook a flight), and a written policy document the agent must follow. The agent has to draw out the right details, stay within policy, and take the correct — often irreversible — actions; a task counts as solved only when the final database state matches the expected outcome, a strict binary reward. We use its two customer-service domains, retail and airline.
We run τ²-bench’s own implementation of a basic ReAct agent with a two-phase THINK → ACT loop, using
gpt-4.1
as the agent model and
gpt-5.5
as the customer-simulator model; 4 trials per task. The only change we make is a block of learned guidelines appended to the system prompt — everything else (policy, tools, scaffold) is stock τ²-bench. Concretely, the learned guidelines are tacked on after the domain policy as a plain markdown list:
You are a customer service agent that helps users according to the policy below. Always follow the policy strictly. Use the provided tools when needed. <policy> # Retail agent policy As a retail agent, you can help users: - cancel or modify pending orders - return or exchange delivered orders - modify their default user address … </policy> ## Guidelines learned from past task attempts (follow these carefully): - Authenticate the user before taking any order-related action. If email is unavailable, verify identity via full name and zip code using the find_user_id_by_name_zip API. - Always confirm all intended changes (modify, cancel, return, exchange) with a clear, structured summary, and require explicit user approval before executing any database-updating action. - When processing a return or exchange, communicate the refund with an itemized breakdown and clarify refund timing — don't wait for the user to ask. - Take one action per step: never combine a tool call with a user-facing message in the same turn, and wait for the tool's result before replying. - Before looking a user up, parse and clean their name — e.g. split a combined handle like `daiki_johnson_9523` into first/last name — and after a couple of failed lookups, state the exact format you need. …We extract guidelines from the agent’s trajectories on τ²-bench train tasks using
gpt-4.1, the same model that powers the agent, then apply them unchanged on the held-out test set.
Guidelines improve capability and reliability
In a prior post, we talked about how different models / tasks require different dosage of guidance. Inline with the idea, we also evaluate different amounts of guidelines provided to the model.
Each guideline carries a support count — the number of distinct training tasks it was extracted from, so higher support means a rule that generalized across more tasks rather than a one-off. We compare three configurations: baseline (no guidelines), sup3 (keep only guidelines with support ≥ 3 — a small, high-confidence core), and lossless (keep every guideline retained after consolidation, i.e. support ≥ 1).
Table 1. Guideline configurations and retained guideline counts by domain.
The metrics we track. Every task is attempted 4 times, and we read those trials two ways:
- Mean Accuracy — the average success rate across the 4 trials. This is the capability axis: how often does the agent get this right?
- Pass^4 — the percentage of tasks that passed across all 4 trials. This measures along the reliability axis and the metric the τ²-bench leaderboard reports. It matters here because a support agent that resolves a refund 3 times out of 4 still fails a quarter of real customers — Pass^4 asks whether you can trust the agent every time, not just on average.
Figure 1. Mean Accuracy and Pass^4 on held-out retail and airline tasks for agents using
gpt-4.1
across four trials per task.
Agent model:
gpt-4.1
(user simulator:
gpt-5.5
).
Table 2. Held-out agent performance using
gpt-4.1
across four trials per task. Parentheses show percentage-point changes from baseline.
Mean Accuracy climbs in both domains — retail +4.4 pts (78.1 → 82.5) and airline +7.5 pts (57.5 → 65.0). Reliability moves further: the strict Pass^4 rises +10 pts in retail (57.5 → 67.5) and +10 pts in airline (35.0 → 45.0). And the gains come from better-informed calls, not longer conversations — we look at conversation length directly in Better performance costs more tokens, not more turns below.
The best amount of guidance depends on the domain
The two domains disagree about how much guidance pays off. In retail, more is better on every metric — the small sup3 core barely beats baseline on accuracy, and you need the full lossless set for the real jump. In airline the picture flips at the reliability end: lossless leads on accuracy (65% vs 60%), but the compact sup3 core (14 guidelines) is the more reliable one, taking Pass^4 to 45% against lossless’s 40%. Echoing the earlier finding that delivery is a dial, not a constant, support count is a knob worth tuning per domain, not a value to hard-code.
Better performance costs more tokens, not more turns
Guidelines aren’t free, but the cost lands where you’d expect — in prompt input tokens, not in a longer, more meandering conversation:
Figure 2. Total LLM cost per conversation by domain and guideline configuration on the full base split.
Table 3. Total LLM cost per conversation (agent and user simulator) on the full base split across four trials. Lower is better.
Does the agent get there by talking longer? To answer that cleanly we compare like for like: the task runs solved
