Stanford researchers have released Memorilla, an open-source memory system that feeds frozen large language models just 16 tokens of distilled information. On the FactKG benchmark, a frozen Qwen3-8B model scored 84.33% accuracy with Memorilla, compared to 80.43% when fed the entire document collection. This result was achieved using 108 times fewer document tokens. The code, training recipes, and checkpoints are available under the Apache 2.0 licence.
System specifications
The system is a 194-million-parameter trainable module. Regardless of the source collection size, it always outputs exactly 16 memory tokens. The encoder (Qwen3-Embedding-4B) and decoder (Qwen3-8B) remain frozen during operation. Deployment requires a Linux environment with CUDA GPUs and Python 3.11. The quickstart setup needs one GPU with 80 GB of memory; the recipes were tested on H100 80GB units. No quantized variants are currently disclosed.
Architecture and workflow
The pipeline operates in three stages. First, a frozen Qwen3-Embedding-4B embeds each document once. New documents do not force the re-encoding of older ones. Second, 16 learned slots attend to the incoming question and the full store using two cross-attention layers with eight heads. Third, these slots are projected into the Qwen3-8B input embedding space and prepended to the question.
Only the memory module trains. The 194,123,776 parameters for the 8B decoder are updated across three stages. Stage 1 uses roughly 1.98 million Wikipedia paraphrase-reconstruction examples. Stage 2 adds about 1.34 million examples from ten question-answering and summarisation datasets. Stage 3 fine-tunes on all eight benchmarks. The full recipe takes approximately 19 hours on four H100 GPUs.
Benchmark performance
All baselines use the same frozen Qwen3-8B model, according to the paper’s main table. Scores are self-reported by the research team.
- FactKG: 84.33% accuracy, versus 80.43% for full context and 80.38% for MemoRAG.
- PersonaMem-v2: 44.60%, versus 30.90% for xRAG, the next best method.
- TriviaQA: 57.00% exact match, just ahead of MemoRAG at 56.60%.
- LaMP-4 / LaMP-7: 18.15 and 48.37 ROUGE-L, both the best scores.
- Average: 49.68, versus 41.92 for xRAG, 40.75 for LLMLingua-2 and 38.02 for Mem0.
Memorilla underperforms on three benchmarks. On PubMedQA it scores 59.00%, while LLMLingua-2 reaches 69.00%. Those abstracts total only 324 tokens, so compression has little to gain there. MemoRAG leads on PersonalizationV4 (67.88% versus 66.65%) and NarrativeQA ROUGE-L (20.51 versus 19.31). MemoRAG used about 690 times and 50 times more document tokens on those tasks respectively.
Compression peaks on NarrativeQA: 16 tokens replace 67,385 document tokens, a 4,212-fold cut. As agent memory, eight memory tokens lift a Qwen3-4B TextWorld agent from 18.6% to 24.2% average success.
Speed comparison
On a single H100 GPU, Memorilla answers in 0.74 seconds with 512 documents (about 128,000 tokens). Full context slows the response from 0.75 seconds to 19.9 seconds, which is 27 times slower. The memory module itself takes about 2 milliseconds.
An optional memory cache precomputes cross-attention keys and values. On a single B200 GPU, a question then takes 8.1 milliseconds with 1 million documents and 33 milliseconds with 4 million. Using torch.compile, the 1 million case drops to 5.0 milliseconds. The cache costs 80 GiB at 4 million documents.
Deployment
Install with uv on Linux, then load MemoryVLLM.from_pretrained. The first run downloads about 25 GB of models. Checkpoints live at memorilla/Memorilla-Qwen3-8B as Stage 1, 2, 3, PV4 and PMv2 folders.
Each checkpoint must run with the decoder it was trained with. Custom data needs parquet files with collection_id, question, answer and documents columns. The research team also released PersonalizationV4: 149 synthetic users, 200 conversations each, under CC BY 4.0.
Comparison with other systems
Benchmark scores come from Memorilla’s paper, which ran every baseline on the same Qwen3-8B. Licences come from each project’s repository.
Approach: Memorilla uses learned latent memory tokens. MemoRAG uses a memory model to guide retrieval. Mem0 uses extracted text memories. xRAG uses 1-token RAG compression.
Doc tokens per query (geo. mean): Memorilla uses 16. MemoRAG uses 784. Mem0 uses 212. xRAG uses 1.
What the LLM reads: Memorilla feeds 16 soft memory embeddings. MemoRAG feeds retrieved text. Mem0 feeds retrieved memory text. xRAG feeds one projected embedding.
License: Memorilla, MemoRAG and Mem0 are Apache 2.0. xRAG is not disclosed.
FactKG accuracy shows Memorilla at 84.33%, MemoRAG at 80.38%, Mem0 at 72.79% and xRAG at 78.98%. PersonaMem-v2 accuracy shows Memorilla at 44.60%, MemoRAG at 28.76%, Mem0 at 27.48% and xRAG at 30.90%. PubMedQA accuracy shows Memorilla at 59.00%, MemoRAG at 67.00%, Mem0 at 24.00% and xRAG at 36.00%. NarrativeQA ROUGE-L shows Memorilla at 19.31%, MemoRAG at 20.51%, Mem0 at 13.80% and xRAG at 16.92%. The average across eight benchmarks is 49.68 for Memorilla, 47.52 for MemoRAG, 38.02 for Mem0 and 41.92 for xRAG.
What it means
For users working with frozen models, the system removes the need to manage massive context windows. It replaces bulk text with a fixed set of 16 tokens that capture the relevant information. This reduces memory usage and speeds up inference significantly. Developers can integrate the module to improve retrieval accuracy without changing the underlying model weights.
