Underdog Saluki 27B, a 7.9 GB build of Qwen3.8-27B, claims to keep tool calling intact at about two bits per weight and beats the full 54 GB model on a 120-task benchmark. The project uses ISTA-DASLab’s GSQ-RCO quantization. We compare the two GGUF files in stock llama.cpp, run a tensor-by-tensor diff of the headers, and check byte-level differences. We serve each file on one GPU, read memory and speed, and score both on the same seeded 120 tasks from the Berkeley Function Calling Leaderboard using BFCL’s own checker with strict and lenient parsing.
Setup and file pins
The notebook downloads and verifies every file by size and SHA-256 before running anything. It records the llama.cpp release, both model files, and fourteen BFCL files. It fixes the sampler for every request because llama-server otherwise falls back to defaults we did not choose. The script detects the platform, an NVIDIA GPU on Linux or an Apple-silicon Mac, and on Colab it ensures the CUDA 12 runtime libraries the prebuilt llama.cpp needs are importable.
The code checks for an NVIDIA GPU on Linux or an Apple-silicon Mac. It creates folders for models, llama, bfcl, and runs. It downloads llama.cpp version b11565, released 2026-10-12, and two model files. The Saluki file is Underdog-Saluki-27B-1.0-IQ2-mix.gguf from ConwayResearch, 7898369152 bytes. The ISTA file is Qwen3.8-27B-GSQ-RCO-IQ2_XS.gguf, 8422841472 bytes. Both are pulled from specific repository revisions and checked against SHA-256 hashes.
The script also downloads data and checker code from the gorilla repository at commit 6ea57973c7a6097fd7c5915698c54c17c5b1b6c8. This includes BFCL v4 data files like simple_python.json and multiple.json, plus the ast_checker.py and other utility scripts. Underdog Bench provides five kinds of tasks, 24 each: simple_python, multiple, irrelevance, parallel, and parallel_multiple. We draw a seeded sample of 2026 instances with a context window of 16384 tokens. The request sampler uses a temperature of 0, top_k of 0, and top_p of 1.0.
System checks
The script prints the platform, machine type, Python version, and glibc version. On Linux, it prints the OS name, GPU name, memory total, and driver version. On macOS, it prints the chip and unified memory in gigabytes. If the CUDA 12 runtime libraries are missing on Colab, it installs nvidia-cuda-runtime-cu12 and nvidia-cublas-cu12. It verifies the llama-server binary can find all required libraries before proceeding.
Download and verification
The download step fetches the llama.cpp tarball and extracts it. It finds the llama-server binary and makes it executable. If CUDA libraries exist, it adds them to the library path and checks for unresolved dependencies. It runs llama-server –version to confirm the build. Then it downloads each BFCL file and checks its size and SHA-256 hash.
What it means
You can run the 7.9 GB Saluki model on a single GPU and compare it directly to the full-sized ISTA quantization. The test uses a fixed set of 120 function calling tasks with strict and lenient parsing rules. This setup ensures you see performance differences without changes in the model or the evaluation method.
