ampixa
// scripts & text · yak tokenizer

How many pieces does it take to say "Kathmandu is the capital of Nepal"?

1.382 tokens/word on Nepali · 2.7× fewer than Llama-3 · #1 lossless of 42 (1.267) · 18 of 19 languages of Nepal

  • ⌃1.382 → v2.0/128k, not "Yak" in general.
  • ⌃#1 of 42 (1.267) → v1.0/128k-phrase: experimental, one token can span several words, not the build you'd deploy.
  • ⌃18 of 19 → lowest-fertility lossless tokenizer — two lossy ones still beat every Yak build on raw Nepali fertility.
  • +missing line: v1.0/128k scores 1.370 on Nepali — better than v2.0/128k, and better on English and Hindi too. It's also the size the (not-yet-upstream) llama.cpp patch was actually tested against.
5 vs 17tokens for नेपालको राजधानी काठमाडौं हो । — Yak v2.0 vs Llama-3, same sentence
1.370tokens/word, Nepali, v1.0/128k — the best plain build, missed by the one-line summary
1.629tokens/word averaged over 14 other languages of Nepal, v2.0/128k (v1.0: 2.505)
0.1–0.2%of v2.0 tokens are bare vowel-sign splinters, down from 6.7–12.8% across v1
// one sentence, set two ways

Llama-3 reads it as seventeen loose pieces. Yak sets it as five.

Top band: Llama-3's real split of the sentence, piece by piece — न · ेप · ालक · ो · … seventeen in all; several are a single vowel sign with no meaning on its own. Bottom band: Yak v2.0's split of the same sentence — नेपालको · राजधानी · काठमाडौं · हो · । — five pieces, one per word plus the full stop. The picture is made only from where each split actually breaks; nothing here is illustrated.

Type that sentence in Nepali — नेपालको राजधानी काठमाडौं हो । — into most language models and watch what happens underneath. The model doesn't see words. It sees tokens, chunks that a tokenizer decided to cut the sentence into, and for a long time the tokenizers everyone used were built by people thinking in English. Devanagari script, with its vowel signs and conjunct consonants and virama marks, wasn't the shape they had in mind. So a Nepali sentence that a person reads as four words gets chopped into far more pieces than an equivalent English sentence. More pieces means more compute, slower generation, a smaller window to think in, and a higher bill — for the same sentence, just because of which script it's written in.

That's the problem Yak is aimed at. It's a byte-level tokenizer, built from scratch, with a pretokenizer that was taught to respect Nepali word boundaries instead of splitting blindly on vowel signs and joiners. Ampixa released seven versions of it on Hugging Face — small and large vocabularies, a plain v1.0, a wider v2.0 that adds text from other languages spoken in Nepal, and two experimental "phrase" variants that let a single token span up to four words.

The numbers are real, but they don't all belong to the same tokenizer, and it's worth being precise about that — precise enough that the correction sits in the headline above, not buried in a footnote. The record for fewest tokens per word of 42 lossless tokenizers tested, 1.267 on Nepali, belongs to the experimental v1.0 128k-phrase build, where one token can span several words. The best plain build on Nepali is v1.0 128k at 1.370, and it also edges out v2.0 128k on English and Hindi. v2.0 128k scores 1.382 on Nepali, still about 2.7 times fewer tokens than Llama-3 needs for the same text, and it is the one built for the wider country: averaged across fourteen other languages spoken in Nepal it needs 1.629 tokens per word against v1.0's 2.505, and it is the lowest-fertility lossless tokenizer on 18 of 19 languages tested.

There is no single winner. There is a record, a best Nepali build, and a best build for Nepal's other languages.

What's next is the obvious thing: train a model on Yak and measure whether fewer tokens per word really means a better or cheaper Nepali model. The tokenizer is done. Whether it was worth building depends on what gets trained on top of it.

What's easy to miss, buried in a results table

Five intact Nepali token blocks under a crown; across a torn line, a pile of green pixel fragments that sweeps into fewer, squarer pieces.

In the v1 64k and 128k builds, 6.7–9.2% of tokens started on a bare vowel sign — a fragment with no meaning on its own, just wherever byte-pair merging happened to land; across the whole v1 family, including the 128k-phrase build that holds the headline record, it reaches 12.8%. In v2.0 that's down to 0.1–0.2%.

It's an old problem wearing a new mechanism. Nineteenth-century Devanagari metal type couldn't decompose the script into clean, isolated letters the way Latin does either — type foundries cast independent sorts for vowel signs (mātrās) that a compositor combined with a base consonant by hand, in a multi-step process, because the diacritic alone was never a complete, meaningful piece. A byte-pair merge algorithm hits the same wall a punchcutter did a century and a half earlier: a vowel sign in isolation is a fragment, not a letter, whether it's cast in lead or cut by a merge rule.

What's still missing

for "the data is public" read
the tokenizer files are public; the training text and the 14-language-of-Nepal corpus are not. Nobody outside Ampixa can rerun the training or check 1.267 / 1.382 / 1.629 from scratch.
for "free to use" read
CC BY-NC — free to use, not free to sell. Commercial use has to be arranged with Ampixa. The benchmark/eval code is MIT; that does not cover the tokenizer files or the corpus.
for "works in llama.cpp" read
a pre-tokenizer patch exists but isn't upstream yet, and it has only been tested on v1.0's word-level sizes — not confirmed for v2.0 or the phrase variants.
for "a model trained on Yak" read
a line on the release page that says "coming soon." No model has been trained on Yak, and no downstream comparison has been run.
stet
the 1.267 / 1.382 / 1.629 headline numbers themselves — they reproduce within the benchmark's own ±0.03 tolerance on the checks that were re-run, which is a narrow baseline-reproduction check, not an independent re-verification of every row in the 42-tokenizer table.
// also in scripts & text

Not yet their own pages

// receipts · every number on this page is in facts.md