A new GPU allocation system increased cluster usage by up to 33 percentage points on identical hardware. The improvement came from changing the order in which allocation decisions are made, not from upgrading the chips.
Every gain below is measured against a standard First-In-First-Out (FIFO) scheduler running the same workloads. Utilization figures are in percentage points. Value figures represent the percentage increase in priority-weighted output.
The specific decision
Telling a system to “keep GPUs busy” is not an actionable instruction. The actual decision is narrower and harder: determining which GPU runs which job, at which timestep, and at what priority. Formally, this is a binary choice for every combination of GPU, job, and timestep. The output is a grid showing every GPU across the scheduling horizon, with a job name in each cell or nothing at all.
Four workload types compete for this grid: training, real-time inference, batch inference, and quantization. They split into two allocation shapes, and that split is where the difficulty lies. Training, batch inference, and quantization are batch-like. Once started, each needs a contiguous block of GPUs held without interruption until the job finishes. Real-time inference is the opposite. It is elastic, driven by a demand curve that changes every timestep, growing and shrinking as traffic does.
Two incompatible shapes competing for the same hardware in the same timestep is the core problem. A second layer of heterogeneity exists inside a single type. For the same base model, training jobs range from a few hours to several days, and from one GPU to dozens.
The cost of FIFO under contention
The comparison point throughout is a FIFO-based scheduler. Real-time inference is served from a fixed reservation. Every other job is placed in arrival order, without regard for priority.
Under the right conditions, that is a reasonable policy. When the cluster has slack, allocation order costs nothing in utilization. Everything fits regardless of sequence, so FIFO and anything more sophisticated fill the same fraction of the pool. Contention is where that ordering cost stops being invisible and starts costing capacity. It then becomes expensive in two separate ways.
The reservation. Real-time inference cannot wait for capacity. The GPUs have to be there the moment traffic needs them. A scheduler that places jobs in arrival order has no mechanism for releasing GPUs during a trough and reclaiming them before the next peak. The only way to guarantee availability is to take each real-time application’s maximum demand for the day and reserve that many GPUs for the whole day. The cost lands in every hour that is not the peak. An application needing six GPUs at midday and two at 4am holds all six for twenty-four hours. The four idle GPUs are unavailable to any batch job for the entire day. They are not being used, and they are not free either. This is why the baseline sits near half the cluster in the two scenarios where reservation dominates: 51.6% in the mixed control and 53.6% in the training-heavy case. Roughly half a pool, with much of the idle half reserved rather than free. This cost is paid whether the cluster is contended or not — contention only makes it visible.
The ordering. Under real contention, which jobs fit at all depends on the order you place them, not just on how much capacity exists. Order is not a tiebreaker applied after the capacity question is settled. Order is a capacity decision. FIFO places each job as it arrives, without weighing what that job is worth and without checking what else still has to fit inside the horizon. High-priority work waits behind whatever asked first, and capacity gets committed in placements that later jobs cannot use.
The two compound. The block held for the day’s maximum real-time demand is off the table for every batch job in the queue, in every hour. Whatever remains is handed out in the order the requests happened to arrive.
It is the GPU equivalent of an airline assigning aircraft to whichever charter called first, then finding nothing left to fly the route that actually pays. GPUs reserved all day for a peak lasting a couple of hours are grounded aircraft from the previous piece in the most literal sense: on standby, earning nothing, unavailable to anyone else.
Across five benchmark scenarios built for genuine contention, the allocator improved both axes at once. Utilization moved from a 52–85% band to a 72–88% band. Priority-weighted value rose between 24.6% and 105.1%, averaging 52%. Every scenario, both metrics, no tradeoff to explain away.
The strongest single case was a training-heavy workload on 8 GPUs. Utilization went from 53.6% to 87.0%, and value more than doubled, up 105%. Thirty-three points of a fixed, already-depreciating asset were recovered by reclaiming reserved standby capacity and placing the rest in priority order. This figure reflects a single baseline ordering.
The allocator removes both behaviors. Real-time demand is treated as a curve rather than a ceiling. It is allocated against demand at each timestep, with batch-like work occupying the troughs. This is bounded by the cap on how many GPUs a real-time job may swap between consecutive timesteps. Batch-like jobs are placed by priority across the whole horizon instead of in the order they arrived. The rest of this piece is how.
Utilization is necessary. Priority is what turns it into value.
Utilization measures occupancy: what fraction of available GPU-time is allocated to something. It carries no information about what that something is worth. One scenario pulls the two apart completely, and the gap runs in a direction that is easy to miss.
In the scale test, 30 jobs across 64 GPUs, FIFO and the allocator produced identical utilization, 44.9% each, and identical throughput, 27 of 30 jobs completed. The allocator delivered 15.9% more priority-weighted value. Every dashboard reads the same. The cluster produced materially different output.
An objective that does not price priority can fill the cluster to exactly the same level, finish exactly as many jobs, and still deliver less. The previous piece argued that occupancy is a poor read on whether a cluster is earning. This is the measured version of that claim.
Writing the problem down
The alternative is not a longer list of heuristic rules. Some constraints only mean anything globally, and no local rule can express them: contiguous blocks, a budget for how much GPU churn is acceptable across the entire horizon, a guarantee that running work is never preempted. To honor those, the problem has to be written down as one thing.
Five constraints define a legal allocation:
- A GPU serves at most one job per timestep.
- Every job respects its demand range, and whatever is already running is inherited and held.
- Batch-like jobs occupy contiguous blocks of GPUs, sized to a power of two.
- Real-time jobs have a hard cap on how many GPUs they may swap between consecutive timesteps.
- A job that has started cannot be interrupted.
The objective function has two terms. Allocating a GPU to a batch-like job earns a reward equal to its priority multiplied by a time-decay weight. Failing to meet real-time demand incurs a penalty proportional to the size of the shortfall.
The relative size of those weights is the entire service-level policy, expressed as one number. The real-time penalty weight is 5 to 10 times greater than the allocation weight. One unit of unmet real-time demand therefore costs what 5 to 10 GPU-timesteps of equal-priority batch work costs. The asymmetry is deliberate, and it means latency obligations are enforced inside the same optimization that places batch work, rather than by a separate autoscaler competing with the scheduler for the same GPUs.
It is also what makes the elastic treatment of real-time demand safe. The allocator can hand a GPU to batch work during a trough because underserving real-time demand later is priced so far above whatever that batch work earns. The penalty, not a static reservation, is what protects availability.
The time weight decays across the horizon for a reason that only makes sense in an online system. By the next scheduling run, new jobs will have arrived. Capacity used now is worth more than capacity promised later.
The allocator that already knows the constraints
The formal model defines what a legal, well-scored allocation looks like. Answering an incoming request is a separate job, and it belongs to a separate component. This is NP-hard combinatorial allocation. The scheduler is re-invoked every time a job arrives, so the decision has to come back in the gap between two API requests. That latency budget is the fixed constraint the architecture is designed around. A heuristic sits on the hot path and the formal model sits behind it as the specification the heuristic is built to satisfy.
That heuristic is not a generic greedy allocator. Its rules are the formal model’s structural constraints, which means every grid it produces is a legal allocation by construction. Not usually valid. Valid by design.
That design, applied across the whole horizon rather than one arrival at a time, is what produces the utilization gain. The allocator sees every queued job before it places any of them. It can hold the free pool in shapes the remaining work can actually occupy. A batch job needing a contiguous block of a given size still has room when its turn comes. Priority decides who gets first claim on that room. FIFO has neither view. It commits capacity to whichever job asked first. A job that arrives later and needs a specific shape may find nothing left that fits. It goes unscheduled and the GPU-hours it would have consumed go unclaimed.
It runs in 1 to 2 milliseconds on the five contended scenarios, and 15 milliseconds at 64 GPUs and 30 jobs. That is fast enough to run on every incoming request.
The system exposes two modes. Fast mode runs the allocator alone and returns its grid. This is the hot path. Full mode uses that grid as a starting point for the formal model, which attempts to improve on it. This is suited to periodic review rather than per-request decisions.
Results
| Scenario | Utilization | Value | Value gain | Latency |
|---|---|---|---|---|
| Mixed control (8 GPUs, 10 jobs) | 51.6% → 72.4% | 7,093 → 10,980 | +54.8% | 1 ms |
| Real-time contention (8 GPUs, 8 jobs) | 75.0% → 80.2% | 3,233 → 4,029 | +24.6% | 1 ms |
| Training-heavy (8 GPUs, 16 jobs) | 53.6% → 87.0% |
