Google has released TorchTPU, an open-source PyTorch backend that runs models natively on Cloud TPUs. The project, announced at the PyTorch Conference on October 20, 2026, was co-developed with the PyTorch team at Meta. It replaces PyTorch/XLA as the recommended tool for new workloads. Developers can now target a TPU using the standard torch.device("tpu") command, placing the hardware on equal footing with GPUs and CPUs within the PyTorch framework.
Key details
- Scope: A software stack comprising four open repositories: TorchTPU, vLLM-TorchTPU, SGLang-TorchTPU, and TorchTitan. The core library is version 0.2.0 on PyPI, requiring Python 3.11 or later, and is licensed under Apache 2.0.
- Hardware: Runs on Google Cloud TPUs. Testing covers generations from v5e, scaled across 8,192 Ironwood (TPU7x) chips. Serving supports FP8, MXFP4, and NVFP4 checkpoints.
- Execution: Offers eager mode for debugging and
torch.compilefor production throughput. - Performance: TorchTitan achieved 52% MFU training on Qwen 3 8B with Ironwood chips (self-reported).
- Optimisation: Out-of-the-box eager mode took 11.49s per step in a fine-tuning scenario, compared to 2.78s for a fully tuned run.
What is TorchTPU?
TorchTPU is an open-source backend from Google and Meta designed to run PyTorch models directly on TPUs. It connects via PyTorch’s PrivateUse1 interface. Tensors remain ordinary PyTorch tensors without subclasses or wrappers. The system runs one process per device and maintains tensor views and in-place mutations.
The previous tool, PyTorch/XLA, relied on recording lazy-tensor graphs. Developers often had to add code to mark compilable regions. PyTorch/XLA also supported only pure SPMD code. TorchTPU handles divergent MPMD execution as well.
How does TorchTPU execute PyTorch code?
The release includes three eager modes:
- Strict (default): Dispatches each operator asynchronously with standard PyTorch semantics.
- Debug: Runs synchronously with tensor bounds checking.
- DeferAndFuse: Fuses runs of eager operators to cut HBM traffic. The TorchTPU paper notes it can be 50% or more faster than Strict.
For peak speed, torch.compile(backend="tpu") captures the graph with TorchDynamo. It lowers through StableHLO to OpenXLA, which emits TPU machine code. Compiled executables are cached across processes, hosts, and job restarts.
Custom Pallas kernels are callable as standard PyTorch functions. Prebuilt Tokamax kernels and Helion tile kernels also work.
Distribution uses the tpu_dist process group. It supports DDP, FSDP2, and DTensor, along with a topology-aware mesh for TPU interconnects.
Performance figures
All figures below are self-reported by Google and partners.
- Training scale: TorchTitan reached up to 52% MFU on Qwen 3 8B. Google reports near-linear scaling up to 8,192 Ironwood TPUs.
- Tuning ladder: In a TorchTitan fine-tuning case, step time fell from 11.49s to 2.78s. DeferAndFuse alone gave 1.15x speedup. Full
torch.compilegave 3.80x. - Model coverage: Google continuously tests about 5,000 Hugging Face models in eager and compiled modes.
- Fleet serving: Zenteiq grew daily output from 1.47B to 5.97B tokens per 1,000 TPU v6e chips. That gain came from llm-d routing on GKE, not TorchTPU alone.
Integrations
- Hugging Face: Transformers v5 deploys to TPU with
device_map="tpu". Diffusers supports eager,torch.compile, and tensor parallelism. - vLLM-TorchTPU: Built with Inferact. Keeps upstream schedulers and API servers, with custom Pallas kernels.
- SGLang-TorchTPU: Built with RadixArk. Keeps RadixAttention prefix reuse.
- Serving features: FP8 KV caches, FP8, MXFP4, and NVFP4 checkpoints. Speculative decoding includes MTP and EAGLE-3.
- TorchTitan: Recipes for Llama 3 70B/405B, Qwen 3 8B/32B, Qwen 3 235B MoE, and FLUX.1 12B.
- Other tools: llm-d 1.0, Ray, TorchRec, VeRL (preview), and LiteRT-Torch for on-device export.
Comparison with other backends
| Feature | TorchTPU | PyTorch/XLA | TorchNeuron | JAX on TPU |
|---|---|---|---|---|
| Vendor | Google + Meta | Google / PyTorch | AWS | |
| Hardware | Cloud TPU (v5e and later tested) | Cloud TPU | Trainium 2, Trainium 3 | Cloud TPU |
| PyTorch device | tpu | xla | neuron | Not PyTorch |
| Backend mechanism | PrivateUse1 | Lazy-tensor graph recording | PrivateUse1 | Not applicable |
| Default execution | Eager (Strict, Debug, DeferAndFuse) | Lazy tensors | Eager plus torch.compile | Not disclosed |
| Compile path | TorchDynamo to StableHLO to XLA | XLA | Neuron Compiler | XLA |
| Distributed APIs | DDP, FSDP2, DTensor, SPMD and MPMD | Pure SPMD only | FSDP, DTensor, DDP, TP | Not disclosed |
| Custom kernels | Pallas, Tokamax, Helion | Not disclosed | NKI | Pallas |
| vLLM / SGLang serving | vLLM-TorchTPU, SGLang-TorchTPU | Not disclosed | vLLM Neuron | vLLM-JAX, SGLang-JAX |
| Status | Open source, v0.2.0 | Being replaced by TorchTPU | Public beta (Neuron 2.33) | Maintained |
| License | Apache 2.0 | Not disclosed | Not disclosed | Not disclosed |
Installation
Install the package on a TPU VM, then pick the tpu device:
# pip install torch_tpu
import torch
device = torch.device("tpu")
x = torch.randn((4, 4), device=device)The repo’s README pins torch_tpu==0.2.0 with the CPU PyTorch wheel index. Without it, pip pulls the CUDA build of torch. Full guides live at torchtpu.readthedocs.io.
What it means
The shift to a native backend removes the abstraction layer that previously separated PyTorch code from TPU execution. Developers no longer need to rewrite logic or manage lazy graphs manually. The ability to use standard eager mode for debugging means the workflow feels more like CPU training, while compiled graphs handle the heavy lifting in production. This lowers the barrier for teams already invested in the PyTorch ecosystem to adopt Google’s hardware without significant refactoring.
