tokenizers v1: encode, decode and scaling, measured
Training on massive datasets, serving many concurrent requests, or repeatedly processing long inputs can starve the model of data because the tokenizer becomes the bottleneck. Version 1 of the tokenizers library addresses this by prioritising performance so that GPUs do not wait for the CPU to finish tokenisation.
This update produces the same token IDs as version 0.23 while delivering speedups of tens of times in many cases. The work relied on existing open source projects like gigatoken, tiktoken, kitoken, tokie, fastokens, wordchipper and ai-tokenizer, which pushed the boundaries of what a fast tokenizer can achieve.
Before this refactor, contributing to tokenizers may not have seemed worth the effort. The goal now is to make it a library worth contributing to. IBM, NVIDIA and the ExecuTorch team provided patches and helped test across a wide range of hardware to broaden platform support.
Results
We show results for the release candidate against other widely used alternatives. The benchmarks cover single-threaded and multi-threaded performance, scaling across threads, per-model comparison, per-language comparison, latency, decoding throughput, memory heap and crate size.
We run these from the tokbench repository and have added a command to rerun the benchmarks on your hardware.
What V1 Is
v1 preserves the output, the API, the vocabulary and the merge ranks. The library remains general across tokenizer families rather than specialising on BPE, so it loads everything v0.23 loaded.
A tokenizer converts text into the list of integers a model reads. The process runs in four stages. Normalisation applies operations such as lowercasing or Unicode normalisation to the raw text. Pre-tokenisation splits the text into smaller pieces called pre-tokens. The model stage turns each pre-token into tokens and maps them to IDs in its vocabulary. Post-processing adds any special tokens the model expects.
The model stage is where most of the work described here happens. Eight of the ten model families measured use byte pair encoding, or BPE. BPE starts from the bytes of a pre-token and repeatedly joins the highest ranked adjacent pair until no ranked pair remains. The ranking is learned when the tokenizer is trained and ships with it, so the same text always produces the same IDs. A merge never crosses a pre-token boundary. The other two families use WordPiece and Unigram.
Key Changes
Each stage was worked on. These are the changes that mattered:
- workspace split: one crate became a workspace.
tk-encodeis the required runtime, andtk-serialize,tk-convertandtk-trainare linked only when an application needs them. - no-alloc model: the merge working set lives in a caller-owned scratch buffer; the loop never touches the allocator.
- bitcannon: the split pattern becomes Boolean operations over bitstreams, using SIMD instructions to find splits instead of a regex engine.
- merge-loop rewrite: the pieces being merged form an intrusive doubly-linked list inside one preallocated buffer, so a merge updates two indices instead of moving data.
- word cache: a thread-local memo maps pre-token bytes to finished ids, so a repeated word is merged once.
- native parallelism: one shared tokenizer encodes from many threads at once; each thread draws its scratch buffer and word cache from its own sub-pool, so threads no longer queue on a single lock.
The Split: Bitstreams Instead Of A Regex
BPE models use a regular expression to split the input text into smaller, easier to process chunks called pre-tokens. Merges happen inside a pre-token and never across the boundary between two of them, so this split decides what the rest of the pipeline sees.
That regular expression is a fixed parameter of the model. It ships with the tokenizer and never changes at runtime, so there is no need for a general-purpose regex engine to interpret it on every encode. An equivalent splitting function can be written by hand, once, for the pattern a given model actually uses.
A hand-written function can then use the SIMD instructions (single instruction, multiple data) of a modern CPU, which apply one operation to many bytes at once and suit UTF-8 text well. bitcannon views the input’s bytes as parallel streams of bits, so boundaries fall out of boolean operations across whole registers instead of a scan that advances one character at a time. It decides 64 bytes per register operation. The same idea drives Parabix for text processing and simdjson for JSON.
This depends on recognising the pattern. A handful of grammars cover most byte-level BPE models, and a tokenizer whose pattern is not among them keeps the regex path and none of this speed-up. That is why the gains vary as much as they do.
The Word Cache
Real text contains many repeated words. Because BPE always produces the same token IDs for a given pre-token, v1 can save the result after processing it once. A thread-local cache maps each pre-token’s bytes to its token IDs, allowing later occurrences to skip the merge process.
Naturally, as the input grows, the number of unique words can grow more slowly than the total number of words. Repeated words then account for an increasing share of the input. New words still appear, which accounts for the occasional misses in the animation below.
Reproduce the shared-prefix result with:
tokbench measure prefix-sharing \
--engine pipeline \
--engine hf-tokenizers \
--compare-to pipeline-no-cache \
--corpus agentic_sweCaching works best when the input contains repeated pre-tokens. Input with few repeated pre-tokens can pay for lookups without receiving many hits.
The Merge Loop
The next major cost comes from the BPE merge loop. For each pre-token, the loop repeatedly finds the highest-priority adjacent pair and merges it. The previous implementation allocated new memory for every call and built a new priority queue for every pre-token.
v1 reuses a scratch buffer owned by the caller, removing those repeated allocations. It stores symbols in a flat array and links adjacent symbols by their positions in that array, which makes updates during merging cheaper. It also processes a batch of pre-tokens in a single model call.
Each candidate pair is also packed into a single 64-bit value, with the merge rank in the high bits. Comparing two candidates is then just comparing two integers, and “no merge here” is the largest possible value, so the loop finds its next merge without a branch.
Method
Small differences in benchmark design can produce large differences in tokenizer performance. We used the following rules to keep the comparison consistent across engines:
- one timing loop: every engine runs the identical loop; no per-engine fast path.
- load excluded: vocabulary load is timed separately, never inside encode.
- id-hash verified: FNV-1a over the output ids must match the baseline exactly.
- common cells only: medians are over cells every engine ran and verified.
- complete sweep per process: each repeat starts in a new process and retains every cell.
- physical-core pinning: workers are pinned to eight distinct physical cores, never sibling SMT threads.
- independent Jobs: separate Jobs measure host-to-host variation.
Repeatedly encoding one document can be faster than encoding a stream of distinct documents on the same build. The first approach measures performance when the entire document is already represented in the cache. The second measures performance on new input while allowing previously seen pre-tokens to remain cached.
Both conditions are sometimes described as “warm,” even though they measure different workloads. Our headline results use distinct documents, and the complete corpus is too large to fit in the cache. Tokenizer benchmarks should identify which workload they use because the choice can dominate the result.
What This Adds Up To
Across the ten model families v1’s encode path covers, it encodes text 3 to 30 times faster than v0.23 with one thread on an Apple M4 Max. The low end is t5-base, the high end gpt2. It scales at 76% of linear across eight workers. Throughout these changes, v1 produces exactly the same token IDs as the released library.
The overall improvement comes from several changes working together: a hand-written splitter in place of a regex engine, a cache that answers a repeated word without merging it again, a merge loop that never touches the allocator, and one model call per batch of pre-tokens instead of one per pre-token. Each reduces the work done at a different point in the pipeline.
The next priority is support for more model families. We will move additional models onto the new merge loop before 1.0.0. Once the release candidates stabilize, the next step will be bringing about the improvements within the transformers library and the rest of the ecosystem which depend on the tokenizers library.
This post is generated from tokbench results and will be updated as support expands.
Getting It
A release candidate for v1 is on crates.io. The API you call is the one you already call, so the only thing that changes is which build you install.
It is the ordinary install:
cargo add tokenizers --preTraining is behind a default-on feature that pulls a C++ dependency with it. If you only need to encode, turn it off to exclude the training implementation:
cargo add tokenizers --pre --no-default-features --features httpEncoding is unchanged: same call, same ids.
use tokenizers::tokenizer::{Result, Tokenizer};fn main() -> Result<()> {
let tokenizer = Tokenizer::from_pretrained("deepseek-ai/DeepSeek-V4-Flash", None)?;
let encoding = tokenizer.encode("The tokenizer is no longer the bottleneck.", false)?; println!("{:?}", encoding.get_ids()); // [671, 17840, 9160, 344, 1119, 5827, 270, 111127, 16] println!("{:?}", encoding.get_tokens()); // ["The", "Ġtoken", "izer", "Ġis", "Ġno", "Ġlonger", "Ġthe", "
Source Read original →
