Gigatoken, a Rust-based tokenizer released by Marcel Rød, a Stanford PhD student, processes text at 24.53 GB/s on a single machine. This figure comes from a benchmark of the GPT-2 tokenizer using the 11.9 GB owt_train.txt corpus on a dual-socket AMD EPYC 9565 with 144 cores. On that same hardware, OpenAI‘s tiktoken managed 36.0 MB/s and HuggingFace tokenizers reached 24.8 MB/s. The difference represents a 681x improvement over tiktoken and a 989x improvement over HuggingFace.
Performance varied by hardware but the gap remained consistent. On an Apple M4 Max with 16 cores, the tokenizer hit 8.79 GB/s, which is 1,268x faster than HuggingFace and 140x faster than tiktoken. A consumer AMD Ryzen 7 9800X3D achieved 6.27 GB/s, or 106x and 68x respectively. The speedup is not an artifact of a single CPU or vocabulary.
What is Gigatoken
The library is written in Rust with Python bindings and ships on PyPI as gigatoken version 0.9.0. It is licensed under MIT. The codebase is 66.2% Rust and 33.3% Python. Benchmarks cover 23 tokenizer families including GPT-2, GPT-OSS, Llama 3 through 4, Qwen 2 through 3.6, DeepSeek V3/R1/V4, GLM 4 and 5, Kimi K2, Nemotron 3, Phi-4, OLMo 2/3, ModernBERT, Gemma and Mistral.
There are two ways to use it. Compatibility mode wraps an existing HuggingFace or tiktoken tokenizer to preserve exact output parity. Marcel noted on Hacker News that this mode delivers roughly 200–300x speedup depending on usage, as it still incurs Python overhead for list creation and string-to-bytes conversion. The native Gigatoken API lets Rust read files directly and is where the published numbers come from.
The interactive benchmark explorer
Every number below is drawn from the repository’s benchmarks section and the pretokenizer optimization log. Users can switch CPUs, walk the optimization history, or estimate how long their own corpus would take.
How it gets there
The gains do not come from a better BPE merge loop. They come from two places that most tokenizers treat as solved.
(1) Pretokenization: Most implementations delegate this to a regex engine. Gigatoken hand-writes it. The optimization log tracks single-threaded GPT-2 pretokenizer throughput on 100 MB of OpenWebText, and the progression is instructive:
- A fancy-regex baseline runs at ~47 MiB/s. A hand-rolled state machine hits ~380 MiB/s. A winnow-combinator implementation with NEON SIMD intrinsics reaches 462 MiB/s.
- Replacing winnow with a direct Iterator, adding a 256-byte class lookup table for O(1) first-byte dispatch, and switching from NEON intrinsics to SWAR (SIMD Within A Register) pushes it to 830 MiB/s. SWAR loads 8 bytes as a u64 and checks all 8 for the letter property with branchless arithmetic. It needs no architecture-specific intrinsics.
- The final step is dual-cursor ILP exploitation, reaching 1,049 MiB/s. The insight is that the bottleneck at ~840 MiB/s was latency, not throughput. Each token’s end position depends on the previous one, a serial chain of roughly 25–27 cycles. Running two independent cursors from a safe split point lets the out-of-order engine interleave both streams across idle execution ports.
Net effect on the pretokenizer alone: 2.27x over the winnow + NEON baseline, and 22.3x over the regex implementation.
(2) Pretoken caching: If a word has been seen before, its encoded tokens are looked up rather than recomputed. The author notes this is hard in practice, because the cache grows quickly and pretoken distributions are long-tailed. On top of that, interactions with Python are minimized and threads are designed to interact minimally with each other.
The optimization log is unusually honest about what failed. A hot/cold split using #[cold] and #[inline(never)] regressed to 580 MiB/s and was reverted, because the inline barrier stopped LLVM from optimizing the combined ASCII and unicode loop. A two-pass classification buffer with SWAR transition counting was algorithmically correct but ran at 354 MiB/s, since the extra memory traffic beat the branch savings. Profile-guided optimization had no measurable effect, because the inner loop is already branchless and the word-boundary branch is data-dependent.
Benchmark Methodology Notes
The comparison is not strictly apples-to-apples. Gigatoken encodes entire un-split files, finding its own document boundaries and parallelizing automatically. In contrast, HuggingFace (encode_batch_fast) is evaluated on the first 100 MB and tiktoken (encode_ordinary_batch) on the first 1 GB, both pre-split on <|endoftext|>. Because the baselines omit caching, their throughput remains uniform. Measurements report the best of three interleaved rounds using fresh processes with parallelism enabled.
Vocabulary types also introduce constraints; SentencePiece tokenizers are only partially optimized. On EPYC, Gemma 1 processes at 2.51 GB/s (7.3x speedup), Gemma 3 at 3.43 GB/s (9.6x), and CodeLlama at 3.47 GB/s (10.0x). Although substantial, these gains are an order of magnitude lower than the headline BPE performance.
An independent reproduction on KrabArena verified the results. On a 4-vCPU Intel Xeon VM (2.20 GHz) with a 174 MB OpenWebText slice, Gigatoken 0.9.0 achieved a median of 277.8 MB/s, outperforming tiktoken 0.13.0 (10.62 MB/s) by 26.2x and tokenizers 0.23.1 (3.33 MB/s) by 83.4x. All trials successfully validated 35,356 documents, confirming the performance trend scales with core count.
What it means
For engineers building pipelines, the practical change is a drop in wall-clock time for large datasets. The native API removes the need for Python to manage file reads and string conversions. This allows Rust to handle the heavy lifting without the overhead that usually limits throughput. For models using SentencePiece vocabularies, the speedup is lower at 7–22x, and WordPiece is unsupported. The compatibility mode offers a middle ground for teams that cannot change their existing tokenizer logic.
