From CUDA to MLX: How K-Search Brings Decades of Kernel Expertise to Apple Silicon

Apple researchers have translated decades of CUDA kernel expertise into high-performance code for Apple Silicon using an automated search system.

The team at Berkeley AI Research (BAIR) extended their K-Search framework to support MLX. This allows the system to take existing, highly optimised CUDA kernels and adapt them for Apple’s M-series chips without rewriting the logic from scratch.

The new approach achieves near-expert level performance on Apple Silicon. It delivers a 0.97x speedup compared to the native MLX Attention kernel. On the Mamba SSM kernel, it reaches up to a 20x prefill speedup over the community mlx-lm implementation.

The hardware gap

Apple’s MLX framework gained significant traction in late 2023. It enables local AI inference on hundreds of millions of MacBooks and Mac Studios without cloud costs. The unified memory architecture is particularly attractive for mid-sized models ranging from 7B to 70B parameters.

However, a performance gap remains. Many kernels critical to the NVIDIA ecosystem, such as paged attention and optimised SSM scan kernels, are absent or un-tuned in MLX. The framework runs models correctly but often leaves significant speed on the table.

How K-Search works

K-Search is an evolutionary kernel optimisation framework developed by Shiyi Cao at UC Berkeley Sky Lab. Given a naive kernel and hardware specifications, it runs an iterative loop. An LLM reasons about which optimisations to try next, a code-writing model generates candidates, and those candidates are compiled and benchmarked on real hardware.

Measurements feed back into the search, refining promising directions while dropping dead ends until performance converges.

The system relies on a Spec, a domain-specific document encoding hardware rules and mathematical constraints. This prevents generated code from hallucinating invalid primitives and ensures candidates compile and run efficiently.

In this project, a single model (Gemini 3.5 Pro Preview) plays both roles: maintaining the reasoning state and writing the kernels. The reasoning half is prompted as a “GPU kernel performance engineer”. It must classify the kernel, rewrite the reference computation in canonical form, map out data layout, and hypothesise the likely bottleneck before proposing any changes.

The persistent reasoning state is called a world model. It is a decision tree where each root-to-leaf path composes a full optimisation plan. Every node is scored with an overall rating between 0 and 10, a confidence level between 0 and 1, and specific impacts on memory bandwidth, register pressure, and compute fit.

As the search runs, it ranks partial plans and expands the most promising ones. If the best score fails to improve for a few rounds, the search backs off to explore an alternative branch.

Example node from the attention kernel:

"action": "Replace the threadgroup-memory softmax reduction
             with a register-only reduction: each SIMD group
             owns 8 query rows and reduces across lanes with
             simd_shuffle_xor, removing a threadgroup_barrier.",
"difficulty_1_to_5": 4,
"impacts": {
  "memory_bandwidth": 8,
  "register_pressure": 4,   // risk: spill if Br > 8
  "compute_hw_fit": 9      // SIMD width 32; keep tile 8x8
},
"overall_rating_0_to_10": 8,
"confidence_0_to_1": 0.7

The original K-Search paper evaluated this strategy on CUDA kernels from FlashInfer. Across GQA decode, MLA decode, MLA prefill, and MoE, K-Search improved more consistently than OpenEvolve and ShinkaEvolve over a 120-iteration budget.

Building an MLX backend

To bring K-Search to Apple Silicon, the team first built a native MLX backend. This involved implementing a full MLX-specific task adapter for K-Search.

• An MLX task backend in k_search/tasks/ handling kernel compilation and execution on Apple Silicon via MLX’s Metal/C++ APIs.
• Updated kernel logic to translate CUDA primitives to their MLX equivalents.
• A translation layer that maps CUDA’s memory models to MLX’s unified memory architecture.

The translation layer is the key innovation. It allows K-Search to take existing CUDA kernels as a knowledge base and adapt them into high-quality GPU kernels for Apple Silicon.

Although the focus is on MLX kernels for Apple Silicon, the method applies to any ecosystem where CUDA expertise is transferable.