Meet Meta’s JiTTests: LLM-Generated Tests Designed to Fail Before Risky Code Lands

Meta has promoted its Catching Just-in-Time Tests (JiTTests) from a research prototype to production infrastructure. The system uses large language models to…

By Vane November 16, 2026 4 min read

Meta has promoted its Catching Just-in-Time Tests (JiTTests) from a research prototype to production infrastructure. The system uses large language models to write executable tests that are designed to fail when a code change introduces a bug. A failing test serves as concrete evidence that a diff altered behavior. In the first half of 2026, the tool caught 74 privacy severity errors at the diff stage. The method is detailed in the FSE 2026 paper Just-in-Time Catching Test Generation at Meta.

Scale and performance

The system operates on Meta’s backend code, which contains hundreds of millions of lines. The research paper analysed 22,126 generated tests. It runs on Meta’s internal infrastructure overnight using spare capacity. The judge models used in the paper include Llama3.3-70B, Gemini 3 Pro, and Claude Sonnet 4. These are not publicly available.

Engineers report a 60% true-positive rate at full scale. This is a self-reported figure. The best result was catching 74 privacy SEVs before they landed in the first half of 2026. Privacy SEV occurrence fell by 20% in the same period. The worst result was during the initial deployment, where the system confirmed only 8 of 41 reported catches, a rate of 19.5%.

What are Catching JiTTests

Catching JiTTests are LLM-generated tests that should fail when a diff introduces a bug. Traditional hardening tests pass when written and guard against future regressions. Catching tests target the change under review before it lands. Meta formalised the details in its 2025 paper Harden and Catch.

Most AI bug tools act like faster human reviewers. They read the diff and leave comments. JiTTests instead execute code and show a concrete behavioural difference. Meta team concedes this takes more computational steps. It argues the approach catches bug classes that LLM or agent review can miss.

The production pipeline

The production pipeline has five stages:

  1. Targeting: Runs overnight on spare capacity. It selects the previous day’s risky diffs, using a mechanism similar to Meta’s Diff Risk Score.
  2. Test generation: Two diff-aware workflows produce candidate tests.
  3. LLM-as-Judge ensemble: Several LLMs score each failing test using the test code, failure trace, diff and inferred intent.
  4. RubFake: The Rule-Based False-Positive Killing Environment adds deterministic checks. Its rules flag patterns like reflection, broken mocks and not-implemented exceptions.
  5. Human-on-the-loop: The system decides whether to contact an engineer, based on the combined ranking.

Dodgy diff versus intent-aware

The dodgy diff workflow assumes every diff is buggy. It treats the diff as a mutant of its parent and generates tests that tell them apart.

The intent-aware workflow first infers what the diff should do from its code, title and summary. It lists risks, builds a mutant for each, and writes tests to kill them. Tests that then fail on the real diff become weak catches.

Benchmark results

The research team analyses 22,126 generated tests. Diff-aware methods produced 4x more candidate catches than hardening tests. They produced 20x more than tests that failed by coincidence.

The intent-aware workflow produced weak catches in 6.4% of its tests and on 7.9% of diffs. Dodgy diff scored 2.5% and 4.0%. Meta’s earlier ACH mutation-guided hardening scored 0.8% and 1.6%. The rule-based and LLM assessors cut human review load by 70%.

FeatureCatching JiTTests, intent-awareCatching JiTTests, dodgy diffACH mutation-guidedTestGen-LLM
Org / first publishedMeta, 2026Meta, 2026Meta, 2025Meta, FSE 2024
GoalCatch bugs in the diff under reviewCatch bugs in the diff under reviewHarden code against specific fault classesImprove existing human-written tests
Generated test shouldFail on a buggy diffFail on a buggy diffPass when generatedPass when generated
Aware of the diffYes, infers intent and risksYes, intent-unawareNoNo
Weak catches per test6.4% (52 of 811)2.5% (41 of 1,621)0.8% (6 of 754)2.0% (89 of 4,499)
Diffs with a weak catch7.9%4.0%1.6%3.6%
False-positive filterLLM judge ensemble + RubFake rulesLLM judge ensemble + RubFake rulesNot disclosedBuild, pass and coverage filters
Public availabilityInternal onlyInternal onlyInternal onlyInternal only

Figures from Table 1 of the JiTTests paper (self-reported, Meta codebase). TestGen-LLM row uses the need not add coverage setting. A weak catch is a candidate failure, not a confirmed bug.

Full scale deployment

The initial deployment was modest. Meta reported 41 candidate catches to engineers. Eight were confirmed true positives, a 19.5% rate. Four of those would have caused serious failures in production.

The October 2026 engineering post reports much stronger production numbers:

  • 589 engineers engaged with JiTTests, 64% of those contacted.
  • Engineers reported a 60% true-positive rate.
  • 74 privacy SEVs were caught at the diff stage in the first half of 2026.
  • Overall privacy SEV occurrence fell 20% in the same period.

Human-on-the-loop

Human-in-the-loop means a person reviews every output. Human-on-the-loop means engineers set criteria and monitor results in aggregate. They intervene at the policy level, not on each test. Meta team rarely need to read test code. They only confirm whether an observed behaviour change was intended.

Can developers use JiTTests

The answer is No. JiTTests is internal Meta infrastructure. Meta has not released code, models or an API. Teams can still copy the pattern: diff-aware mutation-guided test generation, an LLM judge ensemble, deterministic filters and aggregate human oversight.

What it means

The shift to production suggests Meta believes the tool can handle the volume of its codebase without constant human intervention. The drop in privacy SEVs indicates the system is effective at identifying specific risks before they reach users. However, the gap between the initial deployment rate and the full-scale rate highlights the difficulty of tuning these systems for real-world engineering teams. The approach remains internal, meaning other companies must build their own versions to achieve similar results.

Scroll to Top