Google XSE Explained: XLA Serving Engine Serves AI Models Without Python at 6 Million Queries a Second

Google XSE XLA Serving Engine explained — official XSE identity

Image: Google Open Source Blog — the official XLA Serving Engine identity (official product image).

TL;DR: Google has open-sourced XLA Serving Engine (XSE), a C++ runtime that serves AI models without Python in the serving path. Announced on 5 November 2026, it already handles over 6 million queries a second inside Google. Teams train in Python with JAX or PyTorch, export the model, and serve it compiled — 2–5x faster than Google's best Python server in the company's own tests.

📌 Key Highlights

Google open-sourced XSE on 5 November 2026 under the Apache 2.0 licence, as part of the OpenXLA ecosystem.
It enables a "Train in Python, Serve in C++" workflow: models are exported to an XLA SavedModel and run in a C++ runtime with no Python interpreter at serving time.
XSE already serves over 6 million queries a second at Google across YouTube Shorts, Google Ads, Search and parts of Gemini Omni.
Google's own tests show 2–5x the throughput of its best Python server, 30–70% less CPU per prediction and about a quarter of the memory.
It is not for chatbots: large language models should stay on vLLM or SGLang. XSE targets recommendation, vision and embedding models.

What is Google XSE?

XLA Serving Engine, or XSE, is a general-purpose model export and inference runtime that Google released as open source on 5 November 2026. It sits inside the OpenXLA Project, the industry-backed effort that compiles machine-learning computations into native code for different processors. The code is available now under the Apache 2.0 licence, free for anyone to use and adapt.

The problem XSE solves is a familiar one. Python is where almost all AI models are built — it is easy to write, and frameworks like JAX and PyTorch are built around it. But keeping a Python interpreter in the live serving path adds overhead: every request pays for interpretation, and deployments carry a whole Python environment with them. Google's answer is to split the two jobs apart. You still train in Python. You just stop serving in it.

How "train in Python, serve in C++" works

The workflow has three steps. First, an engineer trains a model in Python using JAX (including Flax and Keras 3), with PyTorch support planned through an integration with TorchTPU. Second, the trained model is exported to an "XLA SavedModel" — a portable file containing the model's computation as StableHLO, with weights stored using Orbax checkpointing. Third, the XLA Serving Runtime loads that file, compiles it with the XLA compiler for the target hardware, and serves predictions through simple gRPC APIs — all in C++, with no Python process involved.

Google ships Docker images tested on CPUs, NVIDIA GPUs and Google Cloud TPUs, and XSE supports both single-host and multi-host serving. Because the model is compiled rather than interpreted, the runtime can use all the cores on a machine in one process, and the model's performance — including its kernel optimisations — is preserved from training into serving.

Serving detail	XSE (C++ runtime)	Typical Python server
Serving runtime	Native C++, no interpreter	Python interpreter in the request path
Model format	XLA SavedModel (StableHLO)	Framework checkpoint
Hardware	CPU, NVIDIA GPU, Cloud TPU	Same, with Python overhead on each
Throughput (Google's tests)	2–5x a best-effort Python server	Baseline
Memory per prediction	About a quarter	Baseline
Best for	One-shot inference: recommendations, vision, embeddings	Flexible experimentation and prototyping

Google XSE benchmark chart — XSE throughput versus Python servers under a latency target

Image: Google Open Source Blog — Google's benchmark chart: XSE throughput against Python servers under a latency target.

What the performance numbers actually mean

The headline numbers come from Google's own studies, and they deserve a careful read. For short requests, XSE delivered 2–5x the throughput of the best Python server Google could build, and 8–36x that of a straightforward one, while using 30–70% less CPU time per prediction and about a quarter of the memory. In the published benchmark, at one example per request, XSE processed about 30,100 examples a second against 6,100 for the optimised Python server — roughly 4.9x — with 99% of requests finishing inside 20 milliseconds. These are company measurements on Google's workloads, not independently verified results, so treat them as evidence the approach works at Google's scale rather than a guarantee for your hardware.

That scale is the real story. XSE's components have run inside Google's production infrastructure since 2025, and today the engine serves over 6 million queries a second. Its partners show the range: Arm proved JAX and XLA serving on Arm Neoverse platforms, Intel demonstrated native JAX inference on Xeon processors, and advertising platform Moloco runs over 650 billion inferences a day on XSE.

What XSE is not for

Google is explicit about the limits. XSE is a lightweight, one-shot inference engine: one request in, one prediction out. For large autoregressive language models with long contexts — chatbots and long-form generation — it recommends vLLM or SGLang, which have the memory management those workloads need, such as continuous batching and paged key-value caches. TorchTPU integration, ahead-of-time compilation and C++ data preprocessing are all on the roadmap rather than shipped. If your model is a chatbot, XSE is the wrong tool; if it ranks, recommends, embeds or classifies, it is squarely the target.

Why it matters for developers in India

India has one of the world's largest communities of JAX and PyTorch developers, and the workloads XSE targets — product recommendations, search ranking, image and video understanding, ad targeting — are exactly what Indian e-commerce, fintech, media and SaaS teams run every day. Because XSE is Apache 2.0 licensed, there is no licence fee and no vendor lock-in: a team can start on a CPU or GPU it already owns and only move to TPUs if the workload justifies it. For startups where every millisecond of latency and every rupee of compute matters, removing the Python interpreter from the serving path is a practical saving. And because the exported model is a standard XLA SavedModel, the same artifact can run on different hardware — useful for teams that deploy across cloud and on-premises machines.

Should you care?

✅ Yes, if you serve ML models in production. If your team runs recommendation, vision or embedding models behind an API, XSE is worth a weekend experiment: the quick-start is public, the licence is free, and the design — train in Python, serve compiled C++ — is the direction the whole industry is moving. ⏳ Wait, if you serve chatbots or large language models. XSE openly says those workloads belong on vLLM or SGLang for now. And if your models are still prototypes, Python's flexibility is worth more than serving speed — optimise when the traffic arrives, not before.

FAQs

What is Google XSE?

XSE, or XLA Serving Engine, is Google's open-source C++ runtime for serving machine-learning models without Python. Announced on 5 November 2026 under the Apache 2.0 licence, it lets teams train models in Python with JAX or PyTorch, export them as an XLA SavedModel, and serve them compiled on CPUs, NVIDIA GPUs or Google Cloud TPUs.

How do I try XSE?

The quick-start is in the public repository at github.com/openxla/xla-serving, with Docker images tested on CPU, GPU and TPU. You export a trained JAX model to an XLA SavedModel, then run the C++ serving runtime and call it over gRPC. PyTorch support is planned through TorchTPU integration and is not yet available.

Does XSE replace Python for AI development?

No. Python stays exactly where it is — in training and experimentation, where JAX and PyTorch live. XSE only removes Python from the serving path, after the model is trained. Developers keep writing models in Python; the deployed server just no longer runs an interpreter.

Can XSE serve large language models?

Not well, and Google says so directly. XSE is a lightweight one-shot inference engine with no continuous batching or paged key-value cache. For large autoregressive language models with long contexts, Google recommends vLLM or SGLang instead.

Takeaway: Google ran XSE inside its own products for years before open-sourcing it, and the pitch is simple — keep Python for building models, drop it for serving them. For recommendation, vision and embedding workloads, that trade now comes with a free, tested runtime. If that is what you serve, try the quick-start at github.com/openxla/xla-serving and measure it on your own workload before deciding.

TechByCountry — Daily Tech News, Country by Country

Search This Blog