PrismML has released Bonsai 27B. This is a compressed version of Qwen3.6-27B, not a new model trained from scratch. The original architecture remains intact.
Two variants are available under the Apache 2.0 license. Ternary Bonsai 27B uses weights of -1, 0, or +1, achieving 1.71 bits per weight with a total size of 5.9GB. 1-bit Bonsai 27B uses binary weights of -1 and +1, running at 1.125 bits per weight for a 3.9GB footprint.
Both versions handle multimodal tasks. The split is roughly 24.8B language weights, a 0.46B vision tower, and 2.5B for embeddings and the language model head. The vision tower sits separately at 4-bit (HQQ). Context reaches 262K tokens, made practical because approximately 75% of Qwen3.6-27B attention is linear.
That architecture dictates the compression method.
How the Compression Works
Each weight is a code, with one shared FP16 scale per group of 128. The effective weight is calculated as w_i = s_g · t_i.
A ternary value carries log2(3) ≈ 1.585 bits. One FP16 scale per 128 weights adds 16/128, giving ≈1.71 bits per weight. That is a ~9.4× reduction against FP16. Binary costs 1 + 16/128 = 1.125 bits, a ~14.2× reduction.
The representation runs end to end across the matrix-heavy components. Those are embeddings, attention projections, MLP projections, and the LM head. Only a negligible tail of normalization and scale parameters stays higher precision.
Measured as a true average, the Qwen3.6-27B “4-bit” build (Q4_K_XL) is 5.2 bits per weight. The “2-bit” build (IQ2_XXS) is 2.8. Bonsai also departs from BitNet, which avoids collapse only by pretraining from scratch.
The obvious question is what compression costs in accuracy.
Performance
PrismML evaluated 15 benchmarks in thinking mode, using EvalScope with vLLM on H100 GPUs. Ternary Bonsai 27B retains 94.6% of the FP16 baseline, and 1-bit Bonsai 27B retains 89.5%.
| Variant | True bpw | Footprint | Thinking avg | Density (1/GB) |
|---|---|---|---|---|
| Qwen3.6-27B FP16 | 16.0 | 54GB | 85.07 | 0.051 |
| Qwen3.6-27B Q4_K_XL (“4-bit”) | 5.2 | 17.6GB | 84.99 | 0.155 |
| Qwen3.6-27B IQ2_XXS (“2-bit”) | 2.8 | 9.4GB | 72.73 | 0.199 |
| Ternary Bonsai 27B | 1.71 | 5.9GB | 80.49 | 0.400 |
| 1-bit Bonsai 27B | 1.125 | 3.9GB | 76.11 | 0.530 |
| Category | FP16 | Ternary | 1-bit |
|---|---|---|---|
| Math | 95.33 | 93.40 | 91.66 |
| Coding | 88.74 | 85.96 | 81.88 |
| Knowledge and reasoning | 83.15 | 76.96 | 73.39 |
| Agentic and tool calling | 80.00 | 74.01 | 66.03 |
| Instruction following | 78.47 | 71.77 | 65.74 |
| Vision | 72.61 | 65.19 | 59.57 |
Conventional sub-4-bit builds fail differently. IQ2_XXS falls to 57.5 on AIME26 and 56.4 on LiveCodeBench. It still scores 88.93 on MMLU-Redux, so short-form benchmarks mask the collapse. Gemma-4-31B Q2_K_XL repeats that pattern on a second base model.
Scores alone, however, do not explain the release. Memory does.
Memory is the Binding Constraint
Fitting a phone is stricter than storage numbers suggest. iOS limits a single app to roughly half of physical memory. A 12GB iPhone therefore exposes about 6GB.
The KV cache is the second budget. Only 16 of 64 layers carry a growing full-attention cache, so FP16 costs ≈64 KiB/token. A 262K window costs ≈17.2GB, and a 4-bit KV cache cuts that to ≈4.3GB.
Tolerance is measured. Against its own FP16-KV baseline, Ternary Bonsai shows 0.0011 nats of output forward-KL on MATH-500. Q4_K_XL shows 0.0146.
Peaks follow. At 100K tokens with an FP16 cache, 1-bit peaks at 11.6GB and ternary at 14.7GB. The derived Q4_K_XL row needs ≈25.6GB.
Once a model fits, throughput is the next question.
Throughput and DSpark Speculative Decoding
| Platform | Variant | tg128 | pp512 |
|---|---|---|---|
| M5 Max | Binary | 66.4 | 874 |
| M5 Pro | Ternary | 26.2 | 393 |
| iPhone 17 Pro Max | Binary | 11.0 | 111 |
| H100 (CUDA) | Binary | 104.8 | 2755 |
Generation is memory-bandwidth-bound, so fewer bytes per step means more tokens per second. Prefill is compute-bound and gains less.
PrismML also ships a DSpark drafter trained against the Bonsai 27B target. On an H100 at draft depth k=4, the binary target reaches accepted length τ=3.6. That is 143.8 tok/s, a 1.37× speedup. Verification is lossless, so output stays distribution-identical. On Apple Silicon the drafter is off by default at batch size 1.
Running It
Ternary 27B is the demo repo default. Start the server, or generate directly:
./scripts/start_llama_server.sh # OpenAI-compatible API + chat/vision UI on :8080
./llama-cli -m ./Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-Q2_0.gguf \
--mmproj ./Ternary-Bonsai-27B-gguf/mmproj.gguf -c 0 \
-p "Explain KV cache growth."
mlx_lm.generate --model prism-ml/Ternary-Bonsai-27B-mlx-2bit \
--prompt "Explain KV cache growth."Tool calling uses the standard OpenAI-style tools array:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [{"role": "user", "content": "What is the weather in Lisbon?"}],
"tools": [{
"type": "function",
"function": {
"name": "get_weather",
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
}
}]
}'Source Read original →
