Google TorchTPU Open Source PyTorch on TPU Explained: Run PyTorch Natively on TPUs Without a Rewrite
Image: Unsplash (free Unsplash licence) — Illustrative image: code on a screen
📌 KEY HIGHLIGHTS
- Announced at PyTorch Conference 2026 and published 20 October 2026: TorchTPU is now open source on GitHub, and TPUs join GPUs and CPUs as a native PyTorch device.
- Co-developed with Meta's PyTorch team: debug in eager mode, then switch to torch.compile for production performance — the workflow PyTorch developers already know.
- Works with the tools teams already use: vLLM, SGLang, TorchTitan, Ray, TorchRec and Hugging Face — in Transformers v5, a TPU target is simply device_map="tpu".
- The same announcement shipped llm-d 1.0, with Red Hat, IBM and 500+ contributors, for distributed LLM inference on Kubernetes across TPUs and GPUs.
- Reuters frames the move plainly: after Amazon's TorchNeuron, Google is the second hyperscaler attacking Nvidia's real moat — the CUDA software ecosystem, not just chip speed.
What happened?
For years, Google's TPUs were powerful AI chips with an awkward front door for most developers. The world's most popular AI framework is PyTorch, but TPUs were historically tied to TensorFlow and JAX, and the bridge that did exist — PyTorch/XLA, launched in 2020 — used lazy tensors, forced developers to mark out compilable regions in their code, suffered long compile times and offered no native eager execution. In practice, trying a TPU often meant rewriting working code.
TorchTPU is Google's replacement, built from mid-2025 with the PyTorch team at Meta as a proper native backend rather than a bridge. The PyTorch project's own introduction says existing workloads run out of the box with trivial changes — the example given is changing .to("cuda") to .to("tpu"). Project testing reported by MarkTechPost and TechTimes puts compatibility at about 98% of the nearly 5,000 most-downloaded Hugging Face models, with the stack released under Apache 2.0. Google and the PyTorch team recommend TorchTPU over PyTorch/XLA for new workloads.
Google TorchTPU open source PyTorch on TPU explained — old way vs new way
| What changed | PyTorch/XLA (old) | TorchTPU (new) |
|---|---|---|
| Execution style | Lazy-tensor graphs, no native eager mode | Eager-first debugging, torch.compile for production speed |
| Code changes | Intrusive — mark compilable regions by hand | Minimal — in many cases just the device string, e.g. .to("tpu") |
| Hugging Face | Separate Optimum TPU route | Transformers v5 native: device_map="tpu", with nightly TPU testing |
| Serving & training | Limited ecosystem support | vLLM, SGLang, TorchTitan, Ray, VeRL (preview) and llm-d 1.0 built in |
| Scale evidence | — | TorchTitan validated to 8,000 Ironwood TPUs; up to 52% MFU on Qwen 3 8B across up to 8,192 chips (Google-reported) |
The ecosystem pieces matter as much as the backend itself. The vLLM integration, built with Inferact, brings continuous batching, quantization and multi-chip parallelism with custom Pallas kernels. SGLang, built with RadixArk, targets structured and agentic workloads with RadixAttention cache reuse. For training, TorchTitan provides the distributed recipes, and Ray can now schedule different stages of one pipeline across CPUs, GPUs and TPUs. Google also moved LiteRT — formerly TensorFlow Lite — into the PyTorch ecosystem, so a PyTorch model can be converted for phones and edge devices with a single LiteRT-Torch call, and put a TPU agent suite for onboarding and tuning into early preview.
Image: Unsplash (free Unsplash licence) — Illustrative image: an AI processor concept
Why it matters for developers everywhere
Nvidia's dominance in AI training rests only partly on its GPUs. The deeper lock-in is CUDA — the software layer almost every PyTorch model is written and tuned against — which makes switching chips expensive even when rival hardware is cheaper per calculation. Reuters notes Google is following the same play Amazon ran days earlier with TorchNeuron for its Trainium chips: remove the software barrier, and hardware choice becomes a price-and-availability decision again. Meta's endorsement underlines why this can work: Meta says it cannot be constrained by a single hardware architecture, and it is already running major LLM, ads and recommendation workloads through TorchTPU in production. If you follow the open-model side of this shift, our NVIDIA Nemotron 3.5 open models explained post is the companion read, and our Node.js October 2026 security release how to update guide covers the other big developer update this week.
Should you care?
✅ Care if you train or serve PyTorch models and want a real alternative to GPUs — ⏳ wait if you only run small experiments on a laptop. TorchTPU is open-source software, so evaluating it costs nothing but time, and the eager-plus-compile workflow means your debugging habits carry over. The honest caveats: the headline performance numbers are Google-reported, the VeRL and agent-suite pieces are still in preview, and you still need actual TPU access — through Google Cloud or your own infrastructure — to benefit. For teams already priced out of scarce GPUs, though, this is the most credible "try a TPU this week" moment PyTorch developers have had.
FAQs
What is TorchTPU and what does it do?
TorchTPU is Google's native PyTorch stack for its TPU AI chips, now open source on GitHub. Announced at PyTorch Conference 2026, it registers the TPU as a native PyTorch device alongside GPUs and CPUs, so developers can train, serve and deploy models on TPUs using familiar PyTorch APIs, debugging in eager mode and compiling with torch.compile for production performance.
Can I run my existing PyTorch code on a Google TPU now?
In most cases, yes — that is the central promise. Google and the PyTorch team say existing workloads run out of the box with minimal changes, in many cases just switching the device target to "tpu", and project testing reports about 98% of the nearly 5,000 most-downloaded Hugging Face models passing compatibility checks. Models relying on unusual dynamic shapes may still need work, so test before migrating production jobs.
Who developed TorchTPU with Google?
TorchTPU was co-developed with the PyTorch team at Meta, which uses it for LLM, ads and recommendation workloads and feeds production feedback back to both engineering teams. The wider release involved Hugging Face for Transformers and Diffusers support, Inferact for vLLM, RadixArk for SGLang, and Red Hat and IBM among the 500-plus contributors behind the companion llm-d 1.0 release.
Does TorchTPU work with vLLM and Hugging Face?
Yes. The vLLM backend on TorchTPU supports continuous batching, quantization and multi-chip parallelism for serving, and SGLang support targets agentic workloads. On the Hugging Face side, popular Hub models run natively, Transformers v5 targets a TPU with device_map="tpu", Diffusers supports eager mode, torch.compile and tensor parallelism, and Google runs nightly CI testing of the Transformers suite on TPU hardware.
TAKEAWAY
TorchTPU turns Google's TPUs into a native PyTorch target with open-source code and Meta's backing — the CUDA lock-in just got its second serious challenger in a week.
Follow World Tech on TechByCountry for the next hourly update.
Sources: Google Cloud Blog — TorchTPU is now open source: Run native PyTorch on TPUs (20 Oct 2026, primary) • PyTorch Blog — Introducing TorchTPU (primary) • The Hindu / Reuters — After Amazon, Google targets Nvidia's software lock-in with TorchTPU (21 Oct 2026) • Unite.AI — Google open-sources TorchTPU (Oct 2026). Checked 22 Oct 2026.
Comments
Post a Comment