TL;DR. Most Nepali text-to-speech starts from a model trained on Hindi. Fine-tuning teaches it Nepali words, but some Hindi pronunciation can stay behind: a vowel shifts toward its Hindi sound, or goes missing. Hindi Leak Scorer checks for that word by word. We ran it on Swar by Naamche Labs, an open Nepali voice-cloning model fine-tuned from AI4Bharat's Hindi-trained IndicF5. On 26 new Nepali sentences read in 4 voices, it flagged the key word in 36 of 100 Swar readings and 4 of 104 from our Nepali teacher model. Four words were flagged in every Swar voice; all four are a dropped vowel. These are candidates for a listener, not confirmed errors: we have not yet measured the scorer against blind listeners, and a newer head still flags 5 of 13 readings that a native listener accepted. We release the code and every result as a research preview.
Same sentence, same voice prompt: Swar's नेकपा scores 1.000 against both Nepali references for a dropped vowel and speech recognition hears नेक्पा; our teacher's scores 0.000 and is heard as नेकपा.
Why would Hindi leak into Nepali TTS?
Picture a Nepali voice reading आज नेकपा र अरू दलका प्रतिनिधि बैठकमा आए। The words are right. A speech recogniser writes them down almost exactly. But नेकपा comes out as nekpa, with the vowel after क dropped the way Hindi drops it, not ne-ka-pa. A Nepali listener hears it at once.
Starting from a Hindi base is the practical choice. Hindi has far more speech data, and our neighbours have done good work: Naamche Labs' Swar is fine-tuned from AI4Bharat's IndicF5, and on its own NepTTS-Bench results it reports a 6.19% character error and a 2.92% pronunciation error with a studio voice, the lowest of the systems in its comparison. That is exactly why it is the right model to test: the cost of a Hindi base is that pronunciation learned from Hindi can survive fine-tuning, and recognition error rates do not catch it, because the word is still recognisable.
Can you hear it?
Each row is the same sentence, read in the same voice prompt by Swar and by our Nepali teacher. The first player is the word alone, cut at the aligner's boundaries with 120 ms either side; the second is the whole sentence. The scorer flagged every Swar word here, in all four voices, and none of the teacher's.
| Word | Swar · flagged | Our Nepali teacher |
|---|---|---|
| नेकपा | whole sentence | whole sentence |
| हुनलाई | whole sentence | whole sentence |
| पिढको | whole sentence | whole sentence |
| तलको | whole sentence | whole sentence |
All four are a dropped vowel, and speech recognition hears it: Swar's नेकपा came back as "नेक्पा", हुनलाई as "उनलाई" and पिढको as "पीडको". तलको was transcribed correctly, so the ear and the scorer are the only evidence there.
We picked these four because they were flagged in every Swar voice, in both sentences we wrote for each. They are listening examples, not a sample: Swar was not flagged on हैन, नगरौ, सहमति, बैठक or नैतिक, where its Hindi base was, and only one voice per word is shown. Every reading is in the repo with its scores.
How do you hear a vowel go wrong?
Seven steps, real values: the deletion head gives 1.000 against both Nepali references, which both qualify, so नेकपा is flagged.
The scorer never asks "does this sound Hindi?" in general. It asks a narrower question it can measure: at a vowel where Nepali and Hindi pronunciation differ, does this reading move away from a trusted Nepali reading of the same sentence, in the Hindi direction? Everything below is frozen: no model is trained or tuned on the audio it scores. The sections that follow take one stage at a time, with the numbers from the code.
Where do the test sites come from?
Each word gets two pronunciations. The Nepali one comes from our spoken_nepali frontend, which tries a pronunciation lexicon first, then a suffix-stripping rule, then plain letter-to-sound rules, and records which one it used (high, medium or low confidence). The Hindi one comes from espeak-ng's inc/hi voice. हैन gives Nepali h ʌi̯ n ʌ and Hindi h ɛː n; नेकपा gives n e k ʌ p aː and n eː k p aː.
Both are mapped onto the phone vocabulary of the acoustic model we listen with, so a phone becomes a set of acceptable symbols rather than one label (ʌ, ə and ɐ all count as the same open central vowel). A minimum-edit alignment then lines the two up. Two kinds of difference become test sites:
- Vowel deletion: a Nepali schwa (
ʌ) with no Hindi counterpart, like the vowel after क in नेकपा (Nepalin e k ʌ p aː, Hindin eː k p aː). - Vowel quality: an aligned pair of vowels whose symbol sets do not overlap, like
ʌi̯againstɛː.
Everything else in the word is not tested. This is where most of the 724 word positions drop out.
The messy part: the scorer does not yet use the frontend's confidence. For तरकारी the frontend falls back to low-confidence rules and keeps a schwa after र (t ʌ r ʌ k aː r i), which spoken Nepali does not have. Espeak correctly drops it. That turns a correct Nepali reading into a "deleted vowel", which is exactly the false alarm the newer head raised four times (below).
What counts as a trusted Nepali reference?
The comparison voice is our 24-layer Nepali Pocket teacher reading the same sentence in two other voices, never the query's own prompt. A reference counts only if it passes two checks.
First, a word check by speech recognition. Two Nepali recognisers, our Zipformer and IndicConformer, transcribe it. It passes if IndicConformer's character error is at most 8% (or 15% with word error at most 50%), or both stay under 18% character error on romanised text with word error under 70%. A Hindi recogniser also runs, as an audit only; it never changes the decision.
Second, at each test site, the reference must itself prefer the Nepali path in every one of three crops, with the Nepali vowel clearly present (log emission above −1.5) and a possible alignment through it. A reference that is ambiguous at that vowel is dropped for that site, not trusted.
How is the word found and measured?
Word boundaries come from Meta's MMS forced aligner (the 315M-parameter torchaudio bundle), on romanised text at 16 kHz. Each word is cut three times: tight, ±30 ms and ±60 ms, so that one boundary error cannot decide a result.
The listening is done by two frozen phone recognisers, both wav2vec 2.0 models fine-tuned to output phones: facebook/wav2vec2-xlsr-53-espeak-cv-ft (XLSR-53) and facebook/wav2vec2-lv-60-espeak-cv-ft. For each crop we run CTC forward scoring twice over the word, once along the Nepali phone path and once with the Hindi change applied (the vowel removed, or swapped for the Hindi one). The difference per frame is the margin: positive means the audio fits the Hindi version better.
Each site gets 57 numbers per recogniser:
| Group | Count | What it measures |
|---|---|---|
| Path margins | 18 | Hindi-vs-Nepali margin across the three crops for the query (mean, median, min, max, spread), the reference, and query minus reference; fit of each path; which kind of site |
| Vowel evidence, query | 13 | How strongly the Nepali and Hindi vowel phones fire at the site, how long the site lasts, the neighbouring phones, and vowel energy in any gap the alignment left |
| Vowel evidence, reference | 13 | The same 13 for the Nepali reference |
| Difference | 13 | Query minus reference for those 13 |
Every number is relative to a Nepali reading of the same words, so voice, speed and sentence largely cancel.
What decides a flag?
Two tiny networks, 945 parameters each (57 inputs, 16 hidden units, one output). The deletion head reads the LV-60 features; the quality head reads the XLSR-53 features. Each outputs a change score from 0 to 1, not a probability of a real Hindi error.
They were trained on pairs of the same word where we know what changed:
| Pairs | How the change was made |
|---|---|
| Physical | Real recordings, same speaker: the vowel nucleus cut out, or swapped for a nucleus of a different vowel, with 2 ms fades; the sham is the original nucleus put back |
| Natural | Our teacher asked to make one phonemic vowel edit in the same voice, kept only when the second recogniser confirms the edit happened |
| Model | Our teacher (Nepali) against stock IndicF5 (Hindi-trained) on the same text, kept only where the phonetic evidence qualifies |
Deletion trained on 238 pairs (87 held out for development); quality on 180 (60 held out). हैन and किनकि were kept out of training.
A site is flagged only if its score is at least 0.5 and a fixed set of checks passes on both recognisers: the reference prefers Nepali, the query's margin is positive in every crop, and the Hindi phone is actually present. Below 0.2 is "no change"; in between is "uncertain". Across references, every qualified reference must agree, or the site is uncertain; if none qualifies, the site is unscored. A word is flagged if any of its sites is.
The thresholds were set conservatively, and it shows. On held-out spliced recordings the deletion head caught 35 of 190 removed vowels and the quality head 18 of 166 swapped ones, with 0 false flags on the matching shams.
| Held-out test | Caught | False flags |
|---|---|---|
| Physical deletion | 35 / 190 | 0 |
| Physical quality | 18 / 166 | 0 |
| Natural deletion | 26 / 144 | 3 |
| Natural quality | 11 / 144 | 0 |
| IndicF5 vs teacher, deletion | 26 / 160 | 3 |
The scorer misses most edits and rarely flags a correct vowel: a flag means something, a missing flag means little.
One release choice is not clean: the best-scoring quality head missed the हैन development case, so we rolled back to an earlier frozen quality head that catches it. That is a choice made with a known example, which is why हैन and किनकि do not count as evidence below.
Is there one scorer or two?
Two, and the difference matters. Every number in "What did it find?" comes from the reference-paired scorer described above. We are also testing a newer deletion head that needs no reference: it reads the raw convolutional features of XLSR-53 (5 ms frames) over the word and decides alone. It has 54,655 parameters and catches all six known deletion cases, but it fails our check on short retained vowels (40–60 ms), flagging 13 of 144 correct ones, and in a first human probe it flagged 5 of 13 readings a native listener accepted. It is not used for any result in this post.
What did it find?
Swar: 36 of 100 readings flagged. Our Nepali teacher: 4 of 104. Same text, voices and frozen scorer.
We wrote 26 new Nepali sentences the scorer had never seen (16 revisit words listeners had already reviewed, 10 introduce new ones) and had each model read them in the same 4 voices. Readings that failed the speech-recognition word check were left out.
| System | Key word flagged |
|---|---|
| Swar (Naamche Labs) | 36 of 100 (36.0%) |
| Our Nepali teacher model | 4 of 104 (3.8%) |
The teacher is the control: same sentences, same voices, same scorer.
The scorer, the text and the voice prompts were frozen before any of these readings existed, and no Swar audio was used to build the scorer. We ran Swar's default checkpoint (podvoices_ema.pt, Hugging Face revision 8ca6f1b) through its own text frontend, which rewrites Nepali spelling before synthesis, with the same sampler settings for every system. 4 of Swar's 104 readings were quarantined by the word check and are left out of the count.
Swar's flags cluster: नेकपा, हुनलाई, पिढको and तलको in every voice, and none on हैन, नगरौ, सहमति, बैठक or नैतिक. The teacher's 4 flags have not been reviewed either; one is हैन in one voice, which may be a real slip or a false alarm.
What can it check?
Only 126 of 724 word positions carry a vowel contrast the scorer can test; the rest are not judged.
Only words with a modelled Nepali/Hindi vowel contrast are tested: 126 of 724 word positions in this set. Everything else is not judged, for better or worse. Consonants, rhythm and intonation are not scored.
What Hindi Leak Scorer is not
- Not a measured accuracy. We have no blind-listener labels on new audio yet, so we cannot give precision or recall. The two known real errors we had (हैन and किनकि) were used to choose the release.
- Not free of false alarms. In a first probe, the newer vowel-deletion head flagged 5 of 13 human readings that a native listener accepted, 4 of them on तरकारी. Natural Nepali reductions can look like a deleted vowel. We are fixing that before calling any number a score.
- Not a verdict on Swar. Naamche Labs built a strong open model, and its own benchmarks measure things this does not. A flag is a candidate for a listener. We tested Swar because, by its own benchmarks, it is a leading open Nepali model built on a Hindi base, which is exactly where this kind of leak would show.
- Not prosody. "Sounds Hindi" is often rhythm and intonation. This version only hears vowels.
What comes next?
Blind listening: native speakers label flagged and unflagged words without knowing which system made them, so we can report precision, recall and false alarms with real uncertainty. After that, the scorer could become a training signal that tells a Nepali TTS model where it still sounds Hindi.
Where this sits
Scoring pronunciation against what a phone recogniser expects goes back to Goodness of Pronunciation (Witt and Young, 2000), built for language learners. We point the same idea at a TTS model instead of a learner, and compare against a Nepali reading of the same sentence rather than an absolute threshold. The parts are open models: word alignment from MMS (Pratap et al., 2023), phone evidence from XLSR-53 (Conneau et al., 2020) and wav2vec 2.0 (Baevski et al., 2020). The model we measure is Swar by Naamche Labs (code: Naamche-Labs/swar, weights CC BY-SA 4.0), fine-tuned from AI4Bharat's IndicF5. We did not redistribute its weights or audio beyond the clips above.
Get it
- Code and every result: Ampixa/hindi-leak-detector. Swar's numbers are in
reports/cross_system_swar_v84/report.jsonand the teacher's inreports/cross_system_results_v80/report.json, with per-sentence cases next to each, and the human probe inreports/native_acceptability_adjudication_v90. - Run it:
python -m scripts.detect_nepali_leakage --query QUERY.wav --text 'NEPALI TEXT' --out reports/NEW --check-query-content. It needs the frozen aligner, encoder and ASR models listed in the repo README; they are not bundled yet, so today this reproduces on our machines, not yours. - Status: research preview. No licence file yet.