Perplexity has open sourced Lily, a Rust and Metal inference engine designed specifically for running the Qwen3.6-35B-A3B model on Apple Silicon. It serves as the local runtime for Hybrid Compute in Perplexity Computer. The system is a single-process runtime where a Rust layer loads the checkpoint and manages the generation loop, while an OpenAI-compatible chat-completions API streams tokens. Hand-written Metal kernels handle the heavy lifting. Neither PyTorch nor MLX exists in the execution path. Lily is deliberately narrow, targeting one model on one hardware family because that narrowness is the performance argument.
The standalone demo is public in the pplx-garden repository. It offers a Rust and Metal inference server that handles greedy text generation through a minimal OpenAI-compatible HTTP API. The 4-bit checkpoint is 19.4 GB, meaning an Apple silicon Mac with 32 GB or more of unified memory is the realistic floor. Perplexity’s shipping Hybrid Compute product lists macOS 15+ with a 24 GB minimum and 32 GB for best results.
Why specialize at all
The default Mac stack uses MLX plus MLX-LM. This setup already ships a Qwen implementation with grouped expert work, a fused recurrent Metal kernel, and GQA-aware attention. However, those operations must stay reusable across architectures. Lily gives that up and puts model structure, execution plans, and kernel selection inside one runtime.
Three workload shapes
Qwen3.6-35B-A3B stores 35B parameters and activates roughly 3B per token. A router scores 256 experts and picks eight, alongside one shared expert that sees every token. It also mixes 10 full-attention layers using grouped-query attention with 16 query heads and two KV heads alongside 30 Gated DeltaNet layers. That yields three patterns: uneven expert groups, attention over a growing KV cache, and a fixed-size recurrence.
Prefill: keep weights packed, keep routing on the GPU
The checkpoint uses groupwise affine 4-bit quantization, where every group of 64 weights shares a bfloat16 scale and bias. About 70 GB of bfloat16 weights compresses to 19.4 GB. Metal 4 tensor operations consume bfloat16, so weights must be reconstructed first. Lily does that one tile at a time inside the grouped GEMM, holding results in threadgroup memory and accumulating in FP32, so the expanded array never reaches unified memory. In Perplexity’s ablation, that fusion raised end-to-end prefill 77.4% at a 512-token prompt.
Keeping the routing histogram, prefix scan, scatter and block map inside a single GPU command buffer added 89% at 512 tokens by removing CPU synchronization inside each MoE layer. Moving from 16-row to 32-row tiles with four simdgroups added 13.2% at 2K. A register-resident Gated DeltaNet scan added 5.6%. Expert GEMMs are roughly 90% of prefill time. Long prompts run in bounded chunks so temporary activations do not compete with weights and cache for memory.
Decode: minimize bytes moved per token
Batch-1 decode has almost no weight reuse, so bandwidth sets the ceiling. One recorded step launched 795 kernels forming 555 sequential stages. Lily records real dependencies in a concurrent Metal pass so independent kernels overlap. The selected token is written straight into the next step’s GPU-resident input slot, removing a per-token CPU round trip. Four kernel chains are fused to keep intermediates in registers.
Coalesced cache reads lifted key bandwidth from 33.8 to 47.9 GB/s and value bandwidth from 42.0 to 61.8 GB/s. GQA packing, where four query heads share one threadgroup so each KV row loads once, improved decode 23.8% at 32K. A fixed-block attention layout at 32K and above improved decode 7.7% at 32K, 27.4% at 64K, and 40.2% at 128K.
Results
On one 40-core, 128 GB M5 Max at batch 1, loading identical 4-bit checkpoint bytes against MLX-LM’s fastest direct-generation path across ten lengths from 256 to 128K tokens, Lily averaged 4,156 prefill tokens/s versus 3,388 (1.23x) and 170.0 decode tokens/s versus 126.4 (1.35x). At a 4K prompt and 4K context it reached 5,749.9 and 186.6 tokens/s against 4,737.5 and 140.9. It was faster at every recorded point: 1.12–1.42x prefill, 1.31–1.37x decode. A teacher-forced check across 192 positions put Lily’s perplexity 0.04% higher, with the same top-ranked token 96.35% of the time.
What it means
The practical change for users is a faster local experience on Mac hardware. The engine pushes more tokens per second during both the initial prompt processing and the ongoing generation phase. It achieves this by keeping data on the GPU and reducing the time spent moving information between the processor and the graphics chip. Developers can now access the code to build their own local inference solutions without relying on the standard MLX framework.
