Perplexity Engineering has released a technical account of how it serves embeddings on GPUs, detailing the infrastructure behind the pplx-embed model and the ranking systems used in its Search, Computer, and API products.
The team notes that inference on mature Hopper and Blackwell hardware has largely converged. The real gains come from the runtime and the way requests are managed. This involves CUDA graph management, an async result-tracking abstraction, and a request path written in Rust.
Two traffic patterns, one engine
The service handles two distinct workloads. Batch embedding occurs when building or re-indexing a vector database, where throughput is the priority to minimise cost. Online embedding happens at query time, requiring a short query to be embedded quickly. Scoring sits in between; after vector search, large document batches are ranked, balancing both needs.
The decision was not to build a separate embedding engine. Because embedding models are small Transformers, batch embedding resembles compute-bound prefill, while online embedding often resembles memory-bound decode. The research team reused the prefill and decode kernels from its LLM stack.
Ivy, Tulip and ROSE
Three services handle a request:
- Ivy is a Rust HTTP gateway. It performs CPU-side work such as JSON parsing, tokenisation, input templating, and batch splitting. It translates requests into a custom gRPC protocol. It also splits large-batch requests into chunks and load-balances them across replicas, correcting the load imbalance caused by varying production payload sizes.
- Tulip is the inference server interface. It is a gRPC server built with Rust, tokio, and tonic, handling scheduling and batching before dispatching to the engine.
- ROSE (Runtime-Optimized Serving Engine) implements model inference. It is primarily Python, provides kernels, layers and model definitions, manages CUDA graphs, and exposes a step() function to Tulip.
Why the scheduler is deliberately simple
Tulip picks sequences on a first-come, first-served basis as requests accumulate. This simplicity is justified by measurements showing that for small embedding models at the sequence lengths Perplexity serves, the linear cost of dense layers dominates the quadratic cost of attention. Latency is therefore roughly proportional to token count, not sequence count. Once a batch saturates the GPU, around 512 tokens on a sub-billion-parameter model, packing in more sequences does not improve efficiency.
CUDA graphs and LazyTensors
On small batches, CPU-side kernel launching can outweigh GPU execution. Perplexity builds whole-model CUDA graphs for all embedding models, capturing every launch into a single driver call. Because embedding models are small, the inflection point where GPU work exceeds launch cost arrives at batches of thousands of tokens and tens of sequences. Some attention implementations block full-model graphs by depending on dynamic host-side inputs; Perplexity upstreamed changes to FlashInfer to enable capture.
Graphs must be captured per configuration, so token counts are padded to buckets that are multiples of 64 or 256. That still yields thousands of graphs and multiple minutes of capture per model. The fix is lazy capture: each configuration gets an eager warmup run, then triggers capture and replay on its second hit. This costs p99 latency at startup but spreads minutes of eager work across hours.
The second piece is the LazyTensor, which tracks a page-locked host buffer plus a cudaMemcpyAsync and a CUDA event. Instead of step() blocking on the device, it returns a LazyTensor, letting a Rust async task wait on batch N while the CPU enqueues N+1.
What it means
For developers using pplx-embed or relying on Perplexity Search, the update means faster response times for embedding queries and more consistent performance under load. The team has effectively removed the latency penalty associated with managing GPU graphs for smaller models, ensuring that the cost of setting up the computation does not outweigh the time spent actually processing data.
