IBM research finds memory needs depend on model capability

Scaling an evaluation across eight models, from a 30B dense system to frontier proprietary tools, revealed a single fact: agentic memory is not a feature you switch on. It is a dose you calibrate to the model.

Agentic memory is not a feature you switch on. It’s a dose you calibrate to the model.

IBM Research released ALTK-Evolve, a method that allows an agent to learn from its own past trajectories. The system distils reusable guidelines and injects them back at inference time. There are no weight updates and no human annotation required.

The correct dosage changes by model tier. Strong models with headroom want the full guideline set. Weaker models perform best with a compact core plus per-task retrieval. Saturated models show no measurable gain.

Curated retrieval can be the most accurate and cheapest option. gpt-oss-120b gained +16.1pp task completion at only +5% tokens. Prompt caching keeps even the full guideline set affordable in production.

Capability dictates dosage

Not every model benefits from the same amount of memory. Across eight models spanning the capability spectrum, three recurring patterns emerged.

• Strong models with headroom want the full guideline set. They require every guideline, including rare edge-case lessons. They have the capacity to absorb and apply all of it. DeepSeek-V3.2 (671B MoE) climbed +9.5 percentage points in task completion when given its full self-mined guideline set.

• Smaller or weaker models get drowned by a large guideline set. For these, a tight, high-confidence core plus a handful of task-relevant guidelines retrieved per task works best. gpt-oss-120b (117B MoE) gained +16.1pp with this selective approach. The full guideline set gained less and cost ~50% more tokens.

• Already-saturated models show no measurable gain. We call this the saturated pattern. The label describes what we observed, not a proven cause. The model may already have been near its ceiling on these tasks, the guidelines may not have addressed its remaining failures, or it may not have applied the guidance effectively. GLM-5 (745B MoE) sat here in our runs.

What puts a model into one pattern rather than another isn’t simply parameter count. Benchmark headroom, context-window size, architecture, guideline quality, and task distribution all appear to shape where a model lands. Separating those factors is ongoing work. The practical takeaway holds either way: the right dose of memory depends on the model, and we can calibrate it.

Learning happens around the model, not inside it

“Memory” here doesn’t mean replaying a past transcript. It means a guideline set. These are strategies that worked, mistakes to avoid, and edge cases, distilled from the agent’s own prior trajectories. The loop is straightforward.

• The agent attempts tasks and produces trajectories.

• ALTK-Evolve extracts behavioral guidelines from both its successful and unsuccessful runs.

• It consolidates those guidelines into a reusable set.

• At inference time, the agent receives either the full guideline set or a task-relevant selection of it.

No model weights are updated. The learning loop changes the guidance available to the agent, not the underlying model. This is exactly why it is cheap to adopt and portable across the eight models tested.

Results Across the Spectrum

We evaluated on AppWorld. This benchmark contains 585 multi-step tasks (168

test_normal

+ 417

test_challenge

) across 9 simulated apps (calendars, messaging, payments, and so on). Tasks are scored two ways. Whether the agent fully completes each task is TGC — Task Goal Completion. Whether every variant of a scenario passes is SGC — Scenario Goal Completion. This is a stricter, all-or-nothing bar. Full definitions are in the appendix.

The three configurations we compare

Because the confusing part of any memory study is what’s actually in the context window, we define the configurations up front.

Both memory configurations draw from the same guideline set. It was mined once via the loop described above from AppWorld’s training split only. What changes between them is only how that one set is delivered. The full guideline set injects all of it every step. Curated retrieval delivers a selected subset. Never how the guidelines were produced, and no test-split data ever goes into building it.

The number of guidelines a model mines depends on its own capability. We report configurations by strategy — “full guideline set” vs. “curated retrieval” — rather than by raw counts, which aren’t comparable across models.

The three patterns, in one view

Representative models from the eight-model sweep, measured by task completion (TGC) on

test_normal

:

Figure 1. Representative models in the three observed patterns. Bars show TGC on AppWorld

test_normal

for baseline vs. the best-memory configuration. The x-axis begins at 40% to make differences visible. TGC alone understates the larger SGC gains — see the SGC columns in the table below.

The figure plots TGC to keep it readable. The table adds the stricter SGC metric, where the gains are often larger:

Reading the SGC column, the stricter metric usually moves more than TGC. DeepSeek’s SGC jumps +16.1pp against a +9.5pp TGC gain. Good guidelines especially help an agent clear every variant of a scenario, not just the average case. The effect doesn’t disappear at the top of the range. GPT-5.5 and Opus, both near the ceiling on TGC, still gain +7.2 and +7.1pp SGC respectively. Memory keeps paying off as long as a model has a remaining failure mode to target.

The Cheapest Memory Strategy Can Also Be the Best

A practical concern: injecting a full guideline set inflates every ReAct step’s input. The guidelines are re-sent each turn. Here is what we observed:

Table 1. Average token use per task, accumulated across agent steps, measured against the no-memory baseline.

Two takeaways:

• Curated retrieval keeps cost near baseline. For weaker models, where selection wins on accuracy, it also wins on cost. This is the best of both worlds (+16.1pp TGC at only +5% tokens for gpt-oss-120b). Better performance here does not require

  Source Read original →

  Related reading

  • Tencent Cloud Open-Sources TencentDB Agent Memory v2.0: A Team-Level Memory Hub for AI Coding Agents
  • Building Non-Interactive Agentic Coding Workflows with Moonshot AI’s Kimi CLI, JSONL Streaming, Testing, and Session Memory
  • Google Cloud’s Always-On Memory Agent Replaces RAG and Embeddings With Continuous LLM Consolidation on Gemini 3.1 Flash-Lite