A score that grades clips but can't rank voices
The Arena checks whether a voice said the right words — not whether a Nepali listener liked how it sounded.
The Arena's own judge-profile metadata labels its only evaluator: "status": "uncalibrated for spontaneous Nepali".
5 reports have passed internal consistency checks ("Verified submissions" in the live UI). That means the numbers were self-consistent on import — not that anyone independently reran the evaluation.
Say two sentences out loud: उसको काम राम्रो छ। and उसले खाम पठायो। The first word pair differs by one puff of air — काम (work) against खाम (envelope). A Nepali listener hears the difference instantly. If you're choosing a voice for a Nepali product, that's the kind of thing you need to get right, and the only reliable check is a Nepali speaker listening. That's expensive to do every time, so the obvious move is to build the test once and try to automate it. This page is what happened when Ampixa tried.
Ampixa built the first part seriously, in two pieces that don't automatically line up with each other: a fixed bank of 205 synthesis prompts, and a panel of 180 people who identify as native Nepali speakers, who rated ten TTS systems and two human recordings — 6,962 ratings, with the top four systems' confidence intervals overlapping and inter-rater agreement (Krippendorff's alpha) of only 0.17.
Then came the automation attempt, NepaliMOS — a small model trained to predict what a human rating would have been, on a different, overlapping slice of rating data, not the 180-person panel above. Picking a TTS system isn't about scoring one clip, though. It's about ranking systems against each other, and that is a separate question with its own, separate answer below — the same split that trips up stand-in measurements in other fields too: agreeing with the real thing case by case is not the same as agreeing about which option wins overall. Four evaluations share this project's name. The sources are explicit that they should stay on four separate cards, not one.
The question list every evaluation starts from
sent_* synthesis prompts + 160 chirp_* natural-speech references. Only the 205 prompts feed the Arena submission track.A listening panel's ballot
6,962 ratings · 10 TTS systems + 2 human
95% CI (top 4 systems): overlap
Krippendorff's α: 0.17
One dated snapshot — a separate export reports 164+/5,760+, another 7,249/7,235/7,169. No manifest reconciles them.
Good on one clip, unproven at ranking
Not statistically significant either way. The right conclusion: no demonstrated ranking ability — not "proven backwards."
Does it say the right words?
5 approved reports · 205/205 prompts each
17 workflow tests · synthetic audio, mocked OAuth
profile status: uncalibrated for spontaneous Nepali
A different, legitimate question — intelligibility, not a listener's opinion. Never calibrated against the native panel.
Combined leaderboard
no such card existsRELEASE_SCOPE.md says outright: no row-level manifest currently joins the human panel, the predictor, the Arena and the prompt coverage. Until one does, this card stays blank — on purpose.
- Counts disagree across README, paper and exports (164+/5,760+ vs 6,962/180 vs 7,249/7,235/7,169) with no frozen manifest.
- NepaliMOS ranks systems close to the opposite order from the human panel (ρ ≈ −0.39, not significant) — unvalidated for ranking unseen systems.
- A larger full-state NepaliMOS checkpoint exists publicly but has never been loaded or recalibrated; the changelog and the code loader disagree about which checkpoint is the real default.
- The Arena's only judge, Whisper-small, is never calibrated against native listening, and a separate internal check already found different judge setups disagree with each other.
pip install neptts-eval404s on PyPI — install requires a pinned source checkout.- Full-audio redistribution rights are per-source and unresolved — it is not a blanket free release.
What's next isn't a bigger leaderboard. It's one dated manifest reconciling the counts, a test of whether the larger NepaliMOS checkpoint ranks any better, and a judge calibrated against the native-listener panel that already exists.
No per-system ratings or per-clip pairs are published, so there is no published distribution to plot. The glyphs are this project's own vocabulary; the picture is the key art's own silhouette (forked wave, crowd, robot, cracked crown) — a drawn argument, not invented data.