NVIDIA’s Nemotron Labs released Dory 1.0 4B, a small reasoning model where middle layers form a shared block that runs zero, one, two, three, or four times per token. Each extra pass adds computation without adding parameters, letting one checkpoint run at four different compute budgets. The tutorial builds the specific llama.cpp branch NVIDIA pins, downloads the 3.7 GB GGUF file, and reads the architecture to predict parameter and cache costs. It checks predictions against a running server, measures decode speed at every loop count, tests whether the cheaper last-loop cache changes predictions, and runs a paired GSM8K evaluation to track accuracy, answer length, and time per solved question as loops increase.
Build the pinned fork and record the environment
The code compiles llama-server from the exact commit of the Dory branch pinned on NVIDIA’s model card. Dory support in llama.cpp is currently an open draft pull request, so no released binary can load the model. The script detects the platform and passes the card’s Metal flags on a Mac, or compiles CUDA kernels for the attached GPU alone on Colab. It skips flash-attention kernels, which the card’s instructions disable, keeping a T4 build manageable. The script verifies the checked-out commit matches the pinned one, ensuring every later step traces to a known version of the code.
Download the Q5_K_M GGUF at a pinned revision
The Q5_K_M GGUF file comes from Hugging Face at a fixed repository revision. The script hashes the file before anything else touches it. NVIDIA’s model card refers to the file as dory-q5_k_m.gguf, but the repository names it NVIDIA-Nemotron-Labs-Dory1.0-4B-Q5_K_M.gguf. Copying the card’s command verbatim fails because of this naming difference. Pinning the real name, size, and SHA-256 together catches both a wrong file and a corrupted download. The 3.67 GB file matches the card’s published checksum exactly, so every measurement that follows uses the same bytes NVIDIA published.
Read the architecture from the GGUF
The script inspects the GGUF file to extract three layer groups and a default of four loops. It prints details for dory.block_count, dory.input_layer_count, dory.recurrent_layer_count, dory.output_layer_count, dory.num_loops, dory.recurrent_kv_cache_mode, dory.attention.sliding_window, and dory.context_length. It breaks the layer pattern into symbols for attention layers, marking sliding window with S and full attention with an asterisk. It counts the total attention layers and separates them by type. The code then sums parameters and bytes for the input block, recurrent block, output block, embedding, and LM head. It also tallies the data types used across the tensors.
What it means
Users can now run a single model checkpoint at varying compute levels by changing how many times the middle layers process each token. The tutorial provides the exact code and file references needed to set this up without relying on a released binary that does not yet support the feature. Accuracy and speed metrics are tied directly to these loop counts, showing the trade-off between cost and performance.
