On July 30, 2026, a bug in the Transformers library allowed the T5 decoder to attend to future tokens during training. The issue remained in the main codebase for nearly two months before detection.
Why bugs reach main
The library supports more than 750 model types. Its test suite contains over 88,000 CPU tests on main and 200,000 GPU tests in daily runs. Model tests form only part of the suite; pipelines, training, distributed execution, quantization and attention kernels are also tested.
Integration tests load real checkpoints and exercise the model API on real hardware. Running all of that on every pull request would keep GPUs busy all day, which is not practical. By the time a scheduled run finds a regression, it may already be in main.
The approach is to shrink the model instead. A fixture is a small model built from the real checkpoint’s configuration. It has fewer and narrower layers but every kind of layer present. Tokenizer and processor files remain unchanged. The fixture is published once, then reused. Fast integration tests run against it on a cheap CPU runner. In the CPU study, the experimental UMT5 fixture used 66 MB of weights; the source checkpoint is 1,227 MB.
This also covers very large models. Some do not fit on standard GPU runners, but their fixtures run on CPU.
Model integrators already build tiny test models using Transformers’ tiny-random-* checkpoints. The goal was a common recipe and checks that go beyond loading the model and running a forward pass. The UMT5 bug would have passed a forward pass just fine.
Fixtures will not replace full integration tests. On PRs, they give a chance to catch bugs before they reach main.
Introducing Model Integration Contracts
transformers-mic is a proposed standard for packaging these tests. A Model Integration Contract (MIC) has three parts:
- a fixture: the shrunk model described above, published once and pinned
- a usage path: a runnable Markdown example of something people do with the model, such as generating text or running a training step
- baselines: the outputs recorded when running that example, reviewed by a human
The contract holds if running the usage path on the fixture still produces the baselines.
A contract covers an architecture and a usage path, not a single checkpoint. Compatible checkpoints can share a fixture, while each keeps its own expected results. They are published in huggingface-mic.
For example, the SmolLM contract generates from prompts such as “The capital of France is” and compares its token IDs with a recorded answer. After loading the model and tokenizer, the generation code is simplified to one prompt:
inputs = tokenizer("The capital of France is", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=4, do_sample=False)
new_tokens = outputs[:, inputs["input_ids"].shape[1]:]
print(tokenizer.batch_decode(new_tokens, skip_special_tokens=True))
It also compares training loss, checks that gradients are finite and verifies that an optimizer step changes the weights. See the full usage.md.
A fleet dataset on the Hub pins each MIC to a frozen revision, along with the package versions used to record its baselines. Each CI run records the fleet revision it used, so reproduction is possible later.
On a PR, MICs are picked based on the files the patch changes.
The UMT5 regression
Frozen MICs were replayed over five months of Transformers commits. The small UMT5 fixture flagged the regression from its introduction on July 30, 2026 until its fix on September 28. Label log-probabilities differed from the reviewed baseline by 1.22 and training loss by about 0.20.
One detail mattered: the contract used SDPA, the default attention path. Running only the eager attention variant missed the bug. A fixture only helps if it goes through the same code as the real checkpoint.
The fix was reverted on a proof branch in Transformers’ CI. The check failed with the same differences, and the selected T5 contract’s runner step took 13.4 seconds on CPU, excluding setup.
The replay results and the CI demo have the measurements and reproduction details.
How models are shrunk
The first recipe was simple: make the dimensions smaller and keep fewer layers. That was not enough for mixed architectures. Keeping two layers is no use if both use sliding attention and the full-attention layers disappear.
The recipe now has four steps:
- Shrink the configuration. Start from a real checkpoint and reduce hidden sizes, attention heads and depth. Keep the different layer kinds, including mixtures of sliding and full attention or dense and MoE layers. Some architectures need extra rules to keep their components compatible. The model’s own Transformers tests are also checked; the
ModelTesterintest_modeling_already picks a small config by hand, so its sizes are taken for any width the generic rule does not know by name..py - Keep preprocessing unchanged. Preserve vocabulary, special tokens and input dimensions, and copy tokenizer and processor files byte for byte. Loading and preprocessing are part of what is tested.
- Build and validate the weights. Try seeded random weights and change the inputs to check that the model responds. If it does not, try training it on the contract’s small batch of examples. Reload the saved model and check that outputs, loss and gradients are finite. Reject failed builds.
- Freeze and record. Save the accepted weights and measure repeatable outputs on a known-good Transformers build. Review the baselines for each supported platform and precision, then publish a pinned snapshot. CI reuses it; it does not rebuild the fixture for every PR.
The original weights file format is also kept. Otherwise the conversion code that runs when a user loads the real checkpoint would be skipped. The full recipe has more detail.
The Llama 4 Scout shrink report shows what this looks like on a big model. 108.6 billion parameters become 44.7 million. The vision stack goes from 34 layers to 2 and the language stack from 48 to 4, keeping the checked layer kinds and MoE routing style.
Building MICs with agents
The CLI, tmic, creates the contract and builds the fixture. For a new model, an LLM drafts the usage code and manifest, and CPU and GPU runs measure the candidate baselines. The LLM drafts; it does not approve. Changed expectations require human review.
To try it on your machine, use your source checkpoint in place of org/model:
pip install "transformers-mic[runtime]" "hf-doc-builder @ git+https://github.com/huggingface/doc-builder.git"
hf auth login
tmic run org/model --local-only --fixtures-only --no-history --bill-to ''
It drafts the contract, builds the fixture and records its baselines on your machine, asking you to review each step, and stops before uploading anything. For a new model, the draft comes from an LLM through Inference Providers, billed to your own account (--bill-to ''). The CLI docs cover the individual steps.
Agents were used to build MICs across model families. When they hit an edge case, the recipe itself was fixed rather than the one fixture, with help from model experts. The published models and remaining gaps are in the coverage space.
Random vs Overfit
With random weights, the Whisper fixture barely responded to changes in the audio. It ran, but that made it a poor test of the audio path.
So it was trained on a few tones paired with words such as “red” and “green”. The goal was not to teach speech recognition, just to make audio changes visible in its outputs. In the weight-initialization study, the random Whisper fixtures caught 18 of 24 injected faults. The overfit fixtures caught all 24.
That gives two ways to build the weights:
- Start with random weights and a fixed seed.
- Overfit on a few examples, like “The capital of France is” => “Paris”.
The Whisper result does not settle which is better. Those were synthetic faults, and a real regression caught only by the overfit version has not been found. Both versions missed an assisted-decoding bug that the real Whisper checkpoint exposed.
So random weights are kept when they pass the checks, and overfit is used when better sensitivity or readable answers are needed. Training takes seconds, but it is one more step that can fail to converge, and one more thing to maintain. The comparison has the results for both.
Open questions
As of November 1, 2026, the MIC job in Transformers CI reports failures without blocking PRs. Whether it should block depends on things not yet measured:
- How many real bugs it catches on PRs, not just in replays
- How often it sends false alarms
- How much work baseline reviews take
It is also unknown how many bugs, like the Whisper assisted-decoding one, only show up with real weights, or whether overfitting catches real bugs that random weights miss.
Coverage is still partial, across model families and the APIs people use. When a regression slips through, the missing usage path can be added and replayed on the broken and fixed versions. Existing tests are kept until their coverage is accounted for.
Finally, linking original Hub model pages to their MICs is desired. A developer or an agent using a checkpoint could then start from an example tested on that checkpoint, its environment and its expected outputs.
Want to help? Pick an uncovered model from the coverage space, try the tmic workflow, and contribute a usage path or review a generated contract.
