TL;DR. Yak is an open, lossless tokenizer built for Nepali first. Its phrase build is the #1 lossless tokenizer on Nepali of the 42 we measured, at 1.267 tokens per word, and it also uses fewer tokens than Llama-3 on English. The plain v1 and v2.0 builds sit within a few percent of Llama-3 on English while using 2.7× fewer tokens on Nepali. Yak v2.0 adds most languages of Nepal, with 38% fewer tokens than the next best tokenizer on a 14-language average. We release the tokenizer files under CC BY-NC 4.0, free for non-commercial use. We do not release the training data.
We measured 19 languages of Nepal: 14 from held-out translation data (Athpariya, Dhimal, Gurung, Kulung, Kayort, Lohorung, Limbu, Magar, Newar, Sampang, Sunuwar, Tamang, Tharu Dangaura, Tharu Rana) and 5 on FLORES+. Yak v2.0 is the lowest-fertility lossless tokenizer on 18 of 19.
Yak at a glance.
If someone asked you how many pieces a language model should cut this sentence into, what would you say?
नेपालको राजधानी काठमाडौं हो । ("Kathmandu is the capital of Nepal.")
Four words and a danda, so five pieces feels right. Llama-3 cuts it into 17.
Same sentence, 3.4× fewer tokens. Every Yak token is a whole word.
Look at how Llama-3 handles Kathmandu: क ाठ म ाड ौं. Several of those pieces start with a vowel sign that has been cut away from its consonant. No reader of Nepali would split the word that way, and a model trained on those pieces has to learn to glue them back together before it can learn anything about the word.
Why count tokens at all?
The standard measure is fertility: how many tokens a tokenizer spends per word (Rust et al., 2021). We measure it on the FLORES+ devtest set, which grew out of the FLORES-200 benchmark from the No Language Left Behind project (NLLB Team, 2022), and we encode the whole file at once. Lower is better.
Token counts are not a vanity number. Petrov et al. (2023) showed that the same text, translated, can cost very different numbers of tokens across languages, and that the gap carries straight into price, latency and how much fits in a context window. A Nepali speaker who pays per token, or who fits less text into the same context window, is paying a tax for the tokenizer's design.
What fertility does not tell you is whether a model trained on that tokenizer will be smarter. We come back to this in the honesty section, because we have not tested it yet.
How many tokens does Nepali cost?
Here is Yak v2.0 128k against the frontier tokenizers on Nepali FLORES+ devtest.
Tokens per word on Nepali FLORES+ devtest. Lower is better.
| Tokenizer | Tokens per word (Nepali) |
|---|---|
| Yak v2.0 128k | 1.382 |
| Gemma 3 | 2.270 |
| GPT-4o (o200k) | 2.321 |
| Llama-3 | 3.764 |
| DeepSeek-V3 | 4.291 |
| Qwen 2.5/3 | 6.575 |
| GPT-4 (cl100k) | 6.979 |
| GPT-2 | 10.973 |
Yak uses 40% fewer tokens than GPT-4o. Llama-3 spends 2.7× as many.
The best Nepali number belongs to a different Yak build. yak-128k-v1-phrase, an experimental build whose tokens may span up to four words, scores 1.267. That is the lowest of every lossless tokenizer we measured. Plain v1 128k is 1.370.
"Lossless" matters here, and we want to be exact about it. Two tokenizers in our panel score lower on Nepali than some Yak builds: sai-nep-128k at 1.321 and nepali-bpe at 1.433. Both lose characters: decode what they encode and you do not get your text back. A tokenizer that drops the hard characters will look efficient for the wrong reason, so our harness marks lossy tokenizers with † and keeps them out of every "lossless" claim. The best lossless tokenizer that isn't Yak is sidskarki-64k at 1.445, and it pays for that with 2.409 tokens per word on English.
Can a small vocabulary keep up?
A bigger vocabulary usually buys lower fertility, so a fair question is whether Yak only wins because 128k is big. It doesn't. The smallest Yak, 32k (32,768 IDs), already beats Arkios and every frontier tokenizer on Nepali.
Nepali tokens per word against vocabulary size (log scale). Bottom-left is small and cheap.
| Tokenizer | Vocab | Tokens per word (Nepali) |
|---|---|---|
| Yak v1 32k | 32,768 IDs | 1.586 |
| IndicBERT v2 | 250k | 1.578 |
| Arkios | 65k | 1.688 |
| Llama-3 | 128k | 3.764 |
| GPT-4o (o200k) | 200k | 2.321 |
| Gemma 3 | 262k | 2.270 |
Half of Arkios's vocabulary and 6% fewer tokens. Within 1% of IndicBERT v2 at one eighth of its vocabulary.
Arkios is a recent open Nepali + English tokenizer at 65k, and the Nepali number in its paper is one of the published results our harness must reproduce before it reports anything. At a matched size the gap is wider: Yak v1 64k scores 1.457 against Arkios 65k at 1.688, which is 13.7% fewer tokens. nyayalm-64k sits at 1.602. IndicBERT v2 (Doddapaneni et al., 2023) is 1.578, slightly lower than our 32k, with a 250k vocabulary.
We are careful with the word "beats" here. Yak 32k does not beat every tokenizer on Nepali. IndicBERT v2 is slightly lower with eight times the vocabulary, and some Nepali-only tokenizers go lower on Nepali and then fall apart on English. The claim is narrower and, we think, more useful: at 32k, Yak beats Arkios and every frontier tokenizer on Nepali, and keeps English usable.
How much does English pay for it?
This was the question we expected to lose on. A tokenizer that gives most of its vocabulary to Nepali should be worse at English. Mostly, it isn't.
English tokens per word. The axis starts at 1.20.
Both languages at once. Bottom-left is best.
| Tokenizer | Tokens per word (English) |
|---|---|
| GPT-4o (o200k) | 1.227 |
| Yak v1 128k phrase | 1.232 |
| DeepSeek-V3 | 1.235 |
| IndicBERT v2 | 1.236 |
| Llama-3 | 1.241 |
| GPT-4 (cl100k) | 1.241 |
| Yak v1 128k | 1.291 |
| Yak v2.0 128k | 1.313 |
The phrase build uses fewer English tokens than Llama-3. Only GPT-4o is lower.
The phrase build beats Llama-3, GPT-4 cl100k, DeepSeek-V3 and IndicBERT v2 on English. The plain builds do not: v1 128k (1.291) and v2.0 128k (1.313) are within about 4–6% of Llama-3. We are stating that plainly because it is the easiest claim to stretch. Plain Yak does not beat Llama-3 or GPT-4o on English. It comes close while spending 2.7× fewer tokens on Nepali, and that trade is the whole point.
What about the other languages of Nepal?
Nepal has many languages besides Nepali, and most tokenizers treat them as an afterthought. In v1 they shared one small bucket of the training mix. Yak v2.0, trained from scratch, gives most of them their own share.
On the 14 held-out languages listed at the top, the gap is wide:
Left: mean over 14 held-out languages. Right: how much more the worst-served language costs than Nepali.
| Tokenizer | Mean tokens per word, 14 languages | Worst language vs Nepali |
|---|---|---|
| Yak v2.0 128k | 1.629 | 1.34× (Athpariya) |
| Yak v2.0 64k | 1.843 | 1.48× (Athpariya) |
| IndicBERT v2 (best other) | 2.639 | 2.79× (Limbu) |
| GPT-4o (o200k) | 3.592 | 2.59× (Limbu) |
| Llama-3 | 4.175 | 2.13× (Limbu) |
38% fewer tokens than the next best tokenizer, averaged over 14 languages of Nepal.
The last column is a parity measure in the sense of Petrov et al. (2023): how much more the worst-served language costs than Nepali. With IndicBERT v2, Limbu costs 2.79× as many tokens as Nepali. With Yak, the worst case is Athpariya at 1.34×. Per language, Yak's margin over the best competitor runs from −16% (Magar) to −51% (Athpariya and Lohorung). It also wins on Bhojpuri, Awadhi, Magahi and Santali (Ol Chiki), and on Tibetan it packs 3.26 characters per token against 1.79 for the best other tokenizer.
All 19 languages: Yak v2.0 128k against the best other lossless tokenizer for each.
The 19th language is Sanskrit, where IndicBERT v2 is 3% lower (2.067 vs 2.126).
Is every Nepali letter in the vocabulary?
This one started as a pull request from a teammate, and it changed how we read every other table.
A byte-level tokenizer can encode any character, even one it has no token for. It falls back to raw bytes, and a single Devanagari character becomes 3–4 byte pieces. The text still round-trips and the unknown-token rate stays at 0%, so the usual checks say everything is fine. The new metric asks a blunter question: does the vocabulary contain this character as a whole token?
Every Yak build has a token for all 73 characters of everyday Nepali and all 128 assigned Devanagari code points. Of the 35 other tokenizers, 20 lack a token for at least one everyday Nepali character (often ऊ ऋ ऐ ः ऽ ॥ ॐ), and 33 lack one somewhere in the Devanagari block. "0% unknown" was hiding the gap.
Everyday Nepali characters with no token of their own, and the Devanagari block without a token. † = lossy on Nepali.
Here is the full picture, sorted by how many everyday Nepali characters each tokenizer is missing. "Byte pieces" is the share of tokens that are raw bytes instead of whole characters, on Nepali FLORES+ devtest and on everyday Nepali text.
| Tokenizer | Everyday Nepali characters without a token | Devanagari block without a token | Byte pieces, Nepali FLORES+ | Byte pieces, everyday Nepali |
|---|---|---|---|---|
| Yak (all 7 builds) | 0 | 0 / 128 | 0.02–0.13% | 0.05–0.20% |
| GPT-2 | 72 of 73 (only ा has a token) | 127 / 128 | 93.62% | 93.01% |
| GPT-4 (cl100k) | 56 of 73 | 111 / 128 | 65.98% | 66.04% |
| airavata | 21: ऊ ऋ ऐ ङ झ ञ ँ ः ऽ ॥ ०–९ ॐ | 75 / 128 | 5.00% | 8.39% |
| Qwen 2.5/3 | 19: ङ ँ ः ़ ऽ ू ृ ै ौ ०–९ | 67 / 128 | 57.02% | 57.24% |
| himalaya-64k / 100k / 128k † | 8: ऊ ऋ ऐ ः ़ ऽ ॥ ॐ | 63 / 128 | 2.98–3.21% | 2.27–2.42% |
| DeepSeek-V3 | 7: ऊ ऋ ऐ ओ झ ढ ॐ | 59 / 128 | 10.13% | 10.33% |
| nyayalm-64k | 6: ऊ ऋ ़ ऽ ॥ ॐ | 61 / 128 | 0.35% | 1.00% |
| param-1 | 6: ऋ ः ऽ ॥ ८ ॐ | 58 / 128 | 0.33% | 0.58% |
| sarvam-1 | 5: ऋ ङ ऽ ॥ ॐ | 55 / 128 | 1.65% | 1.44% |
| Llama-3 | 4: ङ ऽ ॥ ॐ | 53 / 128 | 1.52% | 1.43% |
| sai-nep-128k † | 4: ़ ऽ ॥ ॐ | 59 / 128 | 0.00% | 0.00% |
| mistral-nemo, krutrim-2, sarvam-m | 3: ऋ ऽ ॐ | 49 / 128 | 2.72% | 2.72% |
| Arkios | 1: ॐ | 55 / 128 | 0.46% | 0.57% |
| GPT-4o (o200k), gpt-oss | 1: ॐ | 44 / 128 | 0.67% | 0.59% |
| brahmic-131k, param-2 | 0 | 0 / 128 | 0.66%, 0.35% | 0.59%, 4.63% |
| 13 others: bloom, gemma2, Gemma 3, IndicBERT v2, krutrim-1, mt5, muril, nepali-bpe, nepberta, nllb-200, sarvam-30b, sidskarki-64k, xlm-r | 0 | 3–41 / 128 | ≤ 0.60% | ≤ 0.75% |
Only Yak, brahmic-131k and param-2 cover the whole Devanagari block. † = lossy on Nepali. sai-nep-128k has no byte fallback: it turns its four missing characters into unknown tokens and loses them, so its 0% byte pieces is not a win.
A related check measures how often a token starts on a bare vowel sign or virama, cut off from its consonant (the Kathmandu problem from the top of this post). In v1 (64k and 128k) that was 6.7–9.2% of tokens. In v2.0 it is 0.1–0.2%, and it cost nothing in fertility.
Share of Nepali FLORES+ tokens that start on a vowel sign or virama. Lower is better.
Can I check this myself?
Yes, and we would like you to. Before the harness reports a single Yak number, it has to reproduce 14 published fertility numbers of other tokenizers within ±0.03 (GPT-2, cl100k, o200k, Llama-3, Arkios and others). If it can't match what other people already reported, nothing it says about Yak is worth reading. Lossy tokenizers are marked † and kept out of the lossless rankings. Alongside fertility and parity it reports characters per token, round-trip, script coverage and Rényi efficiency, the entropy-based measure from Zouhar et al. (2023).
To load Yak:
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("ampixa/yak-tokenizer", subfolder="v2.0/128k")
# or subfolder="v1.0/128k", "v1.0/32k", ...
To rerun the benchmark (35 public tokenizers plus every Yak build), and to add your own tokenizer to it:
git clone https://github.com/Ampixa/yak-tokenizer && cd yak-tokenizer
uv sync --extra eval
uv run python scripts/eval_panel.py fetch # FLORES+ is gated: accept its terms on Hugging Face first
uv run python scripts/eval_panel.py run # the full panel, Yak included
uv run python scripts/eval_panel.py run --add mine=path/to/tokenizer.json # plus yours
This measures the panel on FLORES+ (Nepali, English, Hindi, Maithili, Bhojpuri, Sanskrit, Awadhi, Magahi, Santali, Tibetan), everyday Nepali, Romanized and Devanagari word lists, robustness probes and script coverage, and writes the results table to docs/RESULTS.md. Code and math are optional: add --code-math to both fetch and run to measure them too.
Two things you cannot rerun, and we want to be upfront about both:
- Training. We release only the tokenizer, not the dataset. The training data is not published, so you can check every output but not rebuild the vocabulary yourself.
- The 14 minority-language held-out sets. They come from text we may not redistribute, so those 14 columns can't be reproduced outside Ampixa.
Everything else in the leaderboard can be reproduced from the repository.
How it was built
The short version, since the engineering doc covers the rest. Yak is a byte-level BPE tokenizer (Sennrich et al., 2016; byte-level as in GPT-2, Radford et al., 2019), so it is lossless by construction: 100% exact round-trip and 0 unknown tokens on every eval text.
The pipeline is six steps: clean → split → count → mix → merge → ship. We count each domain once and then try many mixes from those counts, which is what made it possible to iterate on one ordinary machine: 15 GB of RAM, 8 CPU cores, no GPU. For v2.0 that meant 36.5 GB of cleaned text across 32 domains, 295 M pre-tokens fed to the trainer, 2.46 M distinct pieces, and 129,776 merges.
Six steps on one CPU machine. yak-merge against the Hugging Face trainer on 2 GB of Nepali.
The merge step runs on yak-merge, a BPE trainer we wrote in Rust. It learns the same merges as the Hugging Face tokenizers trainer. On 2 GB of Nepali it finished in 28 s instead of 358 s (12.7× faster) and used 435 MB of memory instead of 4,226 MB (about 10× less). On a 15 GB machine, that memory difference matters.
Two design rules do most of the work for Nepali. First, words stay whole: vowel signs, virama, nukta and ZWJ/ZWNJ never split a word. Second, there is no normalizer inside the tokenizer, so your text is never altered. Every digit is its own token. The 128k vocabulary has 131,072 IDs: 16 special tokens (chat, think, tool call, code infill, mask), 256 raw bytes, the learned merges, and 1,024 reserved slots so new languages or special tokens can be added later without moving any existing ID.
What Yak is not
- Not the best on Sanskrit. IndicBERT v2 is 3% lower (2.067 vs 2.126).
- Not the best on code or math. 3–19% more tokens than Llama-3, by choice.
- Not one build that wins everything. The v1 phrase build is best on Nepali alone. v2.0 is best across the languages of Nepal, and pays a little on Nepali and English for it (1.382 vs 1.370 on Nepali at 128k).
- Not proven to make a better LLM. We have not trained a language model with Yak and compared it against one trained with another tokenizer. Fewer tokens per word is a strong hint and a direct saving in cost and context, but it is a proxy. Zouhar et al. (2023) and Rust et al. (2021) both found links between tokenizer quality and downstream results, and we expect the same here, but we have not measured it. That test is next.
Where this sits
Most of Yak's ideas come from existing work. BPE comes from Sennrich et al. (2016), and the byte-level variant that makes it lossless from Radford et al. (2019). We use fertility as defined in Rust et al. (2021), and the case that tokenizers treat languages unequally, with the parity measure, is from Petrov et al. (2023). Rényi efficiency is from Zouhar et al. (2023). The evaluation text is FLORES+, the open continuation of the FLORES set from NLLB Team (2022). The comparison tokenizers include Llama-3, GPT-4o's o200k_base via tiktoken, IndicBERT v2 and Arkios, whose pre-tokenization analysis is the closest prior work on keeping Devanagari words whole at pre-tokenization. What Yak adds is the specific combination: lossless, whole Nepali words, every Devanagari character in the vocabulary, a real share of the mix for most languages of Nepal, and a harness that has to reproduce other people's numbers before it reports its own.
Get it
- Hugging Face:
ampixa/yak-tokenizer, withv1.0/(32k, 64k, 128k, plus experimental 64k-phrase and 128k-phrase) andv2.0/(128k flagship and 64k). - GitHub:
Ampixa/yak-tokenizer, with the guide, the benchmark harness and the full results. - Files:
tokenizer.jsonfortransformersandtokenizers, plus a tiktoken export. - License: CC BY-NC 4.0 for the tokenizer files: free for non-commercial use with attribution; for commercial use, contact Ampixa Labs. The benchmark code is MIT. The training data is not released.
If you train a model with Yak, or run your own tokenizer through the panel, we would like to see the numbers, good or bad. Open an issue on GitHub.