Researchers from Columbia University and CertiK have released TensorLean, a tool for formally verifying GPU kernels at the PTX level. The system checks that compiled code matches mathematical specifications. Testing covered 149 kernel instances, including every version of FlashAttention from 2 to 4 and DeepGEMM. The tool proved 83 instances correct and identified 38 correctness bugs in the remaining cases.
In this article
Size and hardware
The framework consists of a 1.8K-line PTX parser. Specifications total 10,278 lines against 151,285 lines of kernel source, a ratio near 15:1. It runs on a 16-core workstation and checks PTX compiled for A100, H100 and B200 kernels.
What it actually does
TensorLean is a formal verification framework for GPU kernels at the PTX level. It starts from unmodified source code. A developer writes a tensor-level spec in Lean. TensorLean then proves four properties: no data races, termination, correct output within tolerance, and untouched memory elsewhere.
The verification pipeline
Specifications build on Lean’s Mathlib. Tensor operators follow PyTorch names, so @ means torch.matmul. Each spec has five parts: spec_math, spec_mut, spec_layout, spec_tol and requires. The math stays separate from the memory layout.
The pipeline runs in seven steps:
- Start from unmodified source.
- Write the tensor-level spec.
- Compile to PTX and load a PTX semantics.
- Build a happens-before graph to check races and termination.
- Symbolically execute to get final memory, identical across schedules.
- Lower the spec to PTX-memory level.
- Discharge proofs with Lean’s kernel, Z3 and cvc5.
Scaling techniques
- Concurrency to sequential: Race-freedom proofs on the happens-before graph cover all CTAs, threads and loop iterations symbolically.
- Loop invariants over semirings: Closed forms for tiled, accumulating loops, such as online softmax, need one SMT query per loop.
- Tolerance proofs: Algebraic error is proved zero. Round-off error is bounded separately, then combined.
- Assertions as hints: Developers add
tensorlean_assertin source. Each is proved once, then reused.
Bugs found
The GitHub README lists 38 bugs, all reported upstream. FlashAttention accounts for 23. Liger Kernel has seven, TileOPs three, TileLang two, DeepGEMM two and Triton one.
By category, the site reports 19 memory access errors and 13 numerical bugs. It also counts seven data races, five uninitialized memory reads and five undefined behaviors. There are three algorithm errors, two miscompilations and one deadlock. One bug can sit in several categories.
Of 102 FlashAttention instances, 56 passed all four properties. The other 46 each failed a proof goal on a source bug.
Comparison with other verifiers
| Feature | TensorLean | Volta | ProofWright | PTX in Lean (Koska et al.) |
|---|---|---|---|---|
| Builders | Columbia, CertiK | MSR India, Stanford | Not disclosed | SF Tensor |
| Code level | PTX | PTX | CUDA | PTX |
| Checks against | Lean tensor spec | Reference kernel | Safety + equivalence | Lean proofs |
| Kernels verified | 149 instances, FA-2/3/4, DeepGEMM, Liger | Tutorial reductions, SGEMM, FA-style, TileLang | KernelBench L1, safety for 74% | 3 simple kernels |
| Async ops (wgmma, TMA) | Yes, FA-3 and FA-4 | Out of scope | Not disclosed | Yes |
| New bugs reported | 38 | Races, OOB reads | Not disclosed | Not disclosed |
| Check time | ~10 min median, 50 min max | 0.007 s to ~570 s | ~3 min per kernel | Not disclosed |
The key gap versus Volta is configuration coverage. TensorLean says equivalence checkers unroll loops for one concrete setup at a time. It uses symbolic parameters instead.
What it means
Developers writing high-performance GPU code can now use a formal method to catch logical errors before deployment. The tool verifies that the compiled PTX behaves exactly as the mathematical spec dictates, covering data races and numerical stability. This is useful for teams maintaining large libraries like FlashAttention or DeepGEMM, where subtle bugs in memory handling or floating-point logic can cause silent failures in production.

