Compressing LLMs by treating block removal as an Ising optimization problem

New research from Multiverse Computing reformulates the selection of transformer blocks for removal as a constrained binary optimization problem. This mathematical structure maps directly onto an Ising glass, a disordered spin system with fixed interactions and a set number of “up” spins. The energy of this system serves as a low-cost proxy for how well a pruned model will perform on benchmarks. Researchers can rank vast numbers of candidate configurations without running a single benchmark, handing the most difficult instances to classical and quantum-inspired solvers. At 50% compression of Llama-3.3-70B-Instruct, this approach gains nearly 23 percentage points on the MMLU benchmark compared to the best competing block-removal method.

Why block selection is a many-body problem

Most existing methods score blocks individually using magnitude, sensitivity, or influence heuristics. In physics terms, these are mean-field approaches that treat each block as if its contribution were independent of the others. A related shortcut restricts removal to a single consecutive run of blocks, which limits the search space but discards most possibilities.

Blocks are not independent, just as spins in a real magnet are not. Whether removing block 20 hurts performance depends on whether block 19 or block 24 was also removed. This interaction, or coupling, between decisions means ignoring them leaves quality on the table, especially when removing large numbers of blocks. Searching over combinations while accounting for these interactions is necessary, but the number of combinations grows exponentially. This regime of exponentially large configuration spaces with pairwise couplings is where statistical physics tools become useful.

The idea: turning block selection into energy minimisation

The team attaches a binary variable to each transformer block. A value of 0 means keep the block, 1 means remove it. They then perform a second-order Taylor expansion of the model’s loss with respect to those variables, producing an approximate Hessian matrix. The diagonal of this matrix represents the individual importance of each block. The off-diagonal entries are the pairwise couplings between blocks, capturing the many-body physics that mean-field methods discard.

This reformulation turns the question of which blocks to remove into a clean optimisation task: find the set of M blocks whose removal minimizes the energy

xᵀH⁰x

, subject to removing exactly M of the N blocks. Mathematically this is a constrained binary optimization problem. Physically it is an Ising glass, an all-to-all coupled spin system with conserved magnetization. The fixed number of removed blocks plays the role of a fixed total spin. The key property established is that this energy is a strong proxy for downstream quality. Low-energy states of the spin system correspond to high-performing pruned models. Minimizing energy and maximizing benchmark score become the same search.

Block selection becomes a constrained binary optimization problem, equivalent to finding low-energy states of an Ising glass; each solution specifies which M of N blocks to delete. Right: the coupling variable α inserted into each block’s residual path to build the Hessian. Source: paper Figure 1.

The reason this is practical is cost. The Hessian, the full set of couplings, is computed once from forward and backward passes on a small calibration dataset. After that, evaluating any candidate configuration is a single cheap energy calculation. There is no need to run the actual model, let alone benchmark it. Because the couplings do not depend on the compression target, the same Hessian can be reused to solve for many different values of M.

Solving it: exact when possible, quantum or quantum-inspired otherwise

For most models the configuration space is large but still checkable. Because computing one energy is so cheap, the team brute-forces it on a single GPU, checking up to tens of billions of spin configurations. A few million take seconds. The hardest tractable case here, removing 8 of Llama-3.3-70B’s 80 blocks (about 29 billion configurations), took roughly two days.

Beyond that the exact approach breaks down. This is where casting the problem as an Ising glass pays off a second time. In its equivalent QUBO form, the constraint is absorbed into a penalty term. The exact same task can be handed to highly optimized classical, quantum, and quantum-inspired solvers built for this class of Hamiltonian. These include the machinery of quantum annealing, QAOA, tabu search, and specialized branch-and-bound. An open-source tabu solver reliably reaches the lowest-energy states in seconds, even on the hardest cases verified against brute force. The method scales to models where enumerating configurations is out of the question, using solvers squarely in Multiverse’s domain.

There is a subtle but important point here, running against the usual grain of optimization. Normally a CBO or annealing solver is judged by whether it finds the true ground state. The team does not actually need the ground state. What is needed is a fast way to generate a handful of good low-energy states. This is a far easier bar, which is why lightweight solvers work so well and why the team can afford to run several of them.

Why the whole low-energy spectrum matters

The energy is a strong proxy for quality, but not a perfect one. The single lowest-energy state is not always the best model. This turns out to be a feature, not a bug. Once the Hamiltonian is set up, reading off the ground state and the low-lying excited states is essentially free. This gives a spectrum of high-quality candidate prunings to try rather than one fragile answer. Exploring excited states, not just the ground state, is itself an area of active physics research and maps neatly onto practitioner needs.

A concrete example involves Llama-3.1-8B-Instruct at 16/32 blocks removed. Most of the top states cut blocks toward the end of the model, as prior work would expect. But the 17th excited state is the first to propose removing a block near the beginning of the model. After light retraining, that configuration outperforms the ground state across several benchmarks. This directly disproves the common assumption that the best pruning is one consecutive chunk of middle-or-late blocks. It shows why respecting the full many-body structure of the problem pays off.

Left: which blocks each of the 20 lowest-energy states removes (red = removed). Right: the 17th excited state, which removes an early block, beats the ground state on several benchmarks after retraining. The best model is an excited state, not the ground state. Source: paper Figure 2.

Results

Across Llama-3.1-8B-Instruct, Qwen3-14B, and Llama-3.3-70B-Instruct, the method (CBO) is on par with or better than state-of-the-art block-removal baselines. The gap widens as compression gets more aggressive.

The clearest win is deep compression of Llama-3.3-70B-Instruct, evaluated without retraining. Up to 24 of 80 blocks removed, CBO is roughly on par with block influence. But at 32/80 and 40/80, it pulls decisively ahead. At the deepest setting, there is an almost 23-point MMLU advantage. It beats the baseline on every benchmark tested. For Qwen3-14B at 12/40 removed, CBO leads MMLU by about 10 points. At lighter compression the methods are comparable, which is expected. The couplings matter most when cutting deep.

At 40/80 (50% depth), CBO holds MMLU near 77 while the strongest baseline falls to the mid-50s. Source: paper Table 2.

It generalizes beyond dense transformers

Block removal gets much harder on modern heterogeneous architectures, where different block types are interleaved. The Ising formulation does not care: a coupling is a coupling regardless of what kind of block sits at each site. To stress-test that, the team applied the method to NVIDIA-Nemotron-3-Nano-30B-A3B-FP8. This is a hybrid model that interleaves Mamba2, attention, and mixture-of-experts (MoE) layers in a non-uniform pattern, without any retraining.

Nothing about the formulation assumes a homogeneous stack, so it transfers directly. Removing 2–3 MoE layers or 2 attention layers, CBO finds configurations that beat block influence on AIME25 and GPQA. The results confirm that redundancy in these hybrid models is real but unevenly distributed. Some expert layers are far more disposable than others. The method’s ability to search the coupled configuration space is what locates the good cuts. Even here, the pattern from the dense models holds. The best configuration is often an excited state rather than the ground state.

Why this fits Multiverse Computing

Reframing a messy machine-learning problem as an Ising Hamiltonian, then solving it with the classical and quantum-inspired optimization machinery built for physics, is squarely in Multiverse’s wheelhouse. It is the same instinct that runs through their compression stack. Block removal composes with the rest of that stack, including quantization, low-rank/SVD compression, width pruning, and knowledge-distillation-based healing. It slots into a larger pipeline rather than competing with it.

For full technical details, including the Taylor-expansion derivation, the QUBO mapping, solver benchmarks, calibration-dataset ablations, and complete results tables, read the full paper on Hugging Face. The code is open-sourced at github.com/CompactifAI/Block_removal_through_constrained_binary_optimization.