TL;DR. Rani continues Ampixa's NepaliConformer speech-recognition line. Its greedy decoder has the lowest WER among ranked systems on NepTel 0.2: 33.5 word errors per 100 reference words. Our older NepaliConformer offline scores 39.7 on the same reference key. On a later private evaluation of 127 real Nepali customer-service phone calls, Rani scores 30.6 WER. AI4Bharat IndicConformer performs better on FLEURS read speech. Rani failed the original release gates on old-reference NepTel and exact phone-number capture. A language model improves some scores but contains customer-call text and is not released. Downloading the acoustic weights requires approval.
Ampixa continued training NepaliConformer for phone speech and evaluated Rani on public benchmarks, private calls and exact entity capture. The code, results and public benchmark support independent scoring of the public results. Private references, differences between historical runtimes and evaluation data used during development limit reproduction and interpretation of the other comparisons.

How did NepaliConformer become Rani?
The model family began with NepaliConformer offline and streaming. The offline encoder came from AI4Bharat IndicConformer. Ampixa trained its TDT/CTC heads and Nepali vocabulary for conversational and phone speech. The separate streaming checkpoint used chunk-limited attention and causal convolutions.
- Original release: domain evaluation. The released offline model scored 31.5% WER on a held-out OpenSLR read-speech slice of 500 utterances with human references. WER rose to 35.6% through the telephony chain. NepTel added real phone audio and per-system outputs. Its original and corrected reference keys define separate benchmark versions.
- Public-system comparisons. Tests covered PVSpeech and FLEURS read speech, the fresh natural-speech panel, synthetic phone-band and degraded variants, and exact entity capture. The older offline model's strong natural-speech result did not carry over to severe degradation. Streaming accuracy was evaluated separately.
- Output adaptation. Expanding the tokenizer failed every quality gate. Keeping the original vocabulary and normalizing training labels passed the next pilot. The Rani continuation used a predefined mixture of call and clean DEV scores for checkpoint selection.
- Clean-speech branch: Rani-IndicConformer. A separate full-model IndicConformer fine-tune used its native Nepali tokenizer. Both pilots failed the FLEURS non-regression gate. A later evaluation produced a lower PVSpeech CER than the pilot evaluation; the cause of the difference is unknown.
- Final evaluation. Rani failed the original release gates with both TDT greedy and beam-plus-LM decoding. It ranks first among entrants on human-corrected NepTel. On the later private-call panel, it outperforms the comparators but exceeds the absolute WER ceiling.
The original NepaliConformer release, the selected parent, final Rani and the parallel clean-speech branch are separate checkpoints. The parent was the starting checkpoint for the later output-adaptation pilots. Their public results are linked in the original measurements and model family.
What is Rani, and what did it inherit?
Rani converts Nepali speech into text. Ampixa built it by continuing training from NepaliConformer offline. That model used AI4Bharat's MIT-licensed IndicConformer Nepali encoder with Ampixa's decoder heads and Nepali vocabulary.
The selected Rani checkpoint has a 17-layer Conformer encoder, model dimension 512, approximately 121 million parameters, and hybrid Token-and-Duration Transducer, or TDT, plus auxiliary CTC heads. This model family uses subsampling by 4 and 40 ms encoder frames. The original SentencePiece vocabulary contains 1,024 pieces. The final Rani evaluations explicitly selected the transducer head. CTC is also available, so inference requires an explicit decoder choice after restoring the model.
| Property | AI4Bharat IndicConformer Nepali | Rani |
|---|---|---|
| Encoder lineage | IndicConformer base | Initialized through the older NepaliConformer model |
| Decoder heads | RNNT plus CTC | TDT plus auxiliary CTC |
| Vocabulary in the architecture comparison | Shared multilingual, 5,632 pieces | Nepali-only, 1,024 pieces |
| Adaptation focus | Indic base model | Conversational Nepali and real Nepali customer-service phone calls |
TDT predicts token durations as well as text tokens, changing transducer alignment and decoding. These experiments varied encoder initialization, vocabulary, labels, data selection, augmentation and decoding together, so they do not isolate TDT's effect on speed or accuracy.
The private release card calls the original tokenizer BPE; preparation notes call it unigram. Reproduction uses the original embedded tokenizer, whose identity and piece count are fixed despite the differing descriptions.
Which public systems did we test on read speech?
The public-system sweep compared read-speech recognition under fixed legacy normalization and greedy decoding, with Rani's LM result included separately. PVSpeech contains 100 sentences from four speakers; FLEURS TEST contains 726 sentences. CER and WER are percentages.

| System | PVSpeech CER | PVSpeech WER | FLEURS CER | FLEURS WER |
|---|---|---|---|---|
| Rani (language model; PVSpeech on 93 sentences) | 3.57 | 17.49 | 10.43 | 29.75 |
| Whisper large-v3 | 26.95 | 88.46 | 24.52 | 87.97 |
| Whisper large-v3-turbo | 37.51 | 90.58 | 28.03 | 87.78 |
| Whisper medium | 44.01 | 99.73 | 43.11 | 110.10 |
| Whisper small | 80.13 | 188.06 | 70.76 | 145.64 |
| Community Whisper (repaired config) | 9.10 | 35.94 | 15.85 | 44.64 |
| MMS-1B | 7.92 | 34.08 | 8.80 | 30.81 |
| Seamless M4T v2 | 5.24 | 18.44 | 8.01 | 22.53 |
| Omnilingual 1B v2 | 7.85 | 31.56 | 9.01 | 31.57 |
| Omnilingual 300M v2 | 13.88 | 50.27 | 12.34 | 44.07 |
| AI-Lab IndicConformer | 7.23 | 18.44 | 11.99 | 31.24 |
| AI4Bharat IndicConformer | 9.94 | 22.02 | 7.43 | 23.19 |
| Community wav2vec2 | 15.75 | 58.89 | 20.49 | 65.17 |
| NepaliConformer offline | 5.11 | 23.34 | 12.38 | 39.20 |
| NepaliConformer streaming | 12.68 | 38.99 | 23.74 | 54.66 |
Community Whisper failed strict loading because its published decoder-layer configuration differed from the checkpoint. Its row uses a repaired configuration. Whisper ran with greedy chunked decoding. IndicWav2Vec was measured on historical NepTel but was not evaluated on PVSpeech or FLEURS in this sweep.
With its language model, Rani has the lowest measured PVSpeech CER: 3.57 on 93 sentences. Rani's PVSpeech WER is 17.49. AI4Bharat IndicConformer has the lowest FLEURS CER, 7.43, with WER 23.19. Seamless M4T v2 has the lowest FLEURS WER, 22.53.
The earlier parent evaluation used a checkpoint with a different hash from final Rani. It produced PVSpeech CER 4.25 on 93 sentences, FLEURS CER 11.60, and old-key NepTel WER 43.31. Its paired clean-speech comparisons used matching scorers and denominators.
The full-panel PVSpeech rows differ in panel size and scoring version: 93 versus 100 sentences. Final Rani scores 4.25 with greedy decoding and 3.57 with beam plus LM. The public sweep used legacy normalization; the later matched clean-speech comparisons used their own fixed protocol.
The public-system sweep, parent evaluation and private release evaluation used separate protocols. The public clean-speech summary links the results.
What did natural, phone-band and degraded speech reveal?
The fresh natural-speech panel contains 96 samples from 12 voices, with Chirp/ASR-derived machine references. NepaliConformer offline leads this panel. Neither the Rani parent, final Rani nor Rani-IndicConformer has a comparable fresh evaluation.
| System | Fresh CER | Fresh WER |
|---|---|---|
| Whisper large-v3 | 68.89 | 116.41 |
| Whisper large-v3-turbo | 127.83 | 184.62 |
| Whisper medium | 173.62 | 197.37 |
| Whisper small | 279.82 | 456.35 |
| Community Whisper (repaired config) | 119.96 | 141.15 |
| MMS-1B | 29.11 | 70.00 |
| Seamless M4T v2 | 52.69 | 76.67 |
| Omnilingual 1B v2 | 20.60 | 59.04 |
| Omnilingual 300M v2 | 26.64 | 72.12 |
| AI-Lab IndicConformer | 16.93 | 38.65 |
| AI4Bharat IndicConformer | 15.44 | 40.00 |
| Community wav2vec2 | 43.53 | 90.00 |
| NepaliConformer offline | 5.48 | 14.23 |
| NepaliConformer streaming | 17.78 | 35.90 |
NepaliConformer offline has the lowest fresh CER, 5.48, and WER, 14.23, among the measured systems. No result is available from Rani's planned qualification on a new human-corrected natural-speech set.
The phone-band test applied synthetic 16→8→16 kHz conversion. A separate synthetic panel tested more severe degradation. The following results are CER percentages under the sweep's legacy scorer. No comparable Rani result is available for these conditions.

| System | PV phone-band CER | Fresh phone-band CER | PV degraded CER | Fresh degraded CER |
|---|---|---|---|---|
| Whisper large-v3 | 26.80 | 60.05 | 67.79 | 140.54 |
| Whisper large-v3-turbo | 29.55 | 135.74 | 265.75 | 248.80 |
| Whisper medium | 34.81 | 186.73 | 163.63 | 225.58 |
| Whisper small | 115.49 | 276.53 | 249.91 | 311.79 |
| Community Whisper (repaired config) | 9.70 | 142.43 | 92.79 | 154.23 |
| MMS-1B | 8.45 | 31.46 | 58.00 | 73.18 |
| Seamless M4T v2 | 6.52 | 68.29 | 55.84 | 135.04 |
| Omnilingual 1B v2 | 9.68 | 24.68 | 74.12 | 72.22 |
| Omnilingual 300M v2 | 16.48 | 33.74 | 77.60 | 74.32 |
| AI-Lab IndicConformer | 5.52 | 17.11 | 39.76 | 56.49 |
| AI4Bharat IndicConformer | 8.41 | 15.90 | 32.51 | 51.73 |
| Community wav2vec2 | 17.98 | 46.46 | 91.03 | 98.40 |
| NepaliConformer offline | 5.75 | 6.34 | 53.37 | 56.67 |
| NepaliConformer streaming | 13.48 | 19.86 | 85.06 | 80.55 |
NepaliConformer offline also has the lowest fresh phone-band CER, 6.34, and WER, 15.38. AI4Bharat IndicConformer leads degraded PVSpeech with CER 32.51 and WER 58.49. The older offline model's degraded fresh CER rises to 56.67.
Error rates can exceed 100% when insertions outnumber reference units. The sweep used pointwise 95% cluster-bootstrap intervals without adjustment for multiple comparisons. The rankings use point estimates. Machine-reference errors, limited speakers and development exposure constrain their interpretation.
The public-system sweep evaluated natural speech, synthetic phone-band audio and degraded audio under the same legacy scoring convention.

Training stages and gate results
Training targeted clean speech, natural speech and calls. The acceptance criteria required improvements over public comparators, a new human-corrected natural-speech evaluation, no private-call regression, and complete capture of at least 81 of 298 phone numbers. The selected parent failed the final checks.
Training combined human-labelled read and natural speech, conversation with machine or pseudo labels, and real Nepali customer-service phone calls. Public sources included OpenSLR, IndicVoices-R and FLEURS train. Later stages applied a shared label convention: digits became spoken Nepali number words, mapped Latin-script English became Devanagari, and punctuation was removed. Labels that could not be mapped or encoded, or that failed length rules, were dropped.
Pilot A used a call-heavy mix and failed the clean-speech gate. At step 3576, private-call DEV WER improved from 48.94 to 40.14. PVSpeech DEV CER improved from 6.00 to 5.95. FLEURS DEV CER worsened from 13.91 to 15.09, a +1.18 percentage-point regression against an allowance of +0.5. No full run followed.
Pilot B used a clean-majority mix and passed. It restarted from the older baseline. The gate still required call DEV WER to improve by at least 2.0 points, with neither clean CER regressing by more than 0.5 points. At step 5024, call WER improved by 8.72, PVSpeech CER by 0.67, and FLEURS DEV CER by 0.73 points.
The full continuation made three passes and ended at step 38712. Selection minimized private-call DEV WER and chose step30000, with WER 34.07 and FLEURS DEV CER 12.89. The final endpoint had call WER 34.27 and FLEURS DEV CER 12.94.
A missing restore-archive dependency caused an export failure and a restart from the preceding checkpoint. Resume handling required corrections to the sampler and epoch counter. Model and optimizer state were retained; exact replay of the historical sample sequence was not guaranteed.
Both stages missed GPU-utilization targets. The full continuation reached 43.26% steady utilization against a 70% target. Later preparation reached 72.34% against an 80% gate. Training continued with that configuration and unchanged recognition-quality gates.
The selected parent failed final checks with PVSpeech CER 4.25, FLEURS TEST CER 11.60, old-key NepTel WER 43.31, and phone capture 65/298. Private-call TEST WER 30.68 passed the non-regression check. The initial PVSpeech decode batch failed before producing usable counts and was retried.
Pilot 2B expanded the tokenizer and failed every quality gate. The vocabulary grew from 1,024 to 2,048 pieces to admit Latin and digit-bearing labels. Preparation aimed to preserve old token identities and compatible tensors while expanding the heads, relocating blank and duration outputs, and resetting optimizer state for changed shapes. Save-and-restore checks passed.
The expand the vocabulary approach was dropped for this run. The gate evaluation gives the following changes relative to the parent:
| Metric | Recorded regression, percentage points | Allowed regression |
|---|---|---|
| Private-call DEV WER | +65.93 | +0.5 |
| PVSpeech DEV CER | +95.23 | +0.3 |
| FLEURS DEV CER | +87.11 | +0.3 |
A later summary lists 65.9 and 95.2 as endpoint scores, while the gate evaluation labels them as deltas. Their interpretation remains uncertain. The pilot failed and no continuation followed. Insufficient adaptation of the changed output system was a proposed explanation.
Pilot 2C kept the original vocabulary and normalized training labels. An initial attempt was cancelled without a completed result or resumable endpoint. The restarted pilot passed all fixed absolute gates:
| DEV metric | Parent | Endpoint | Required upper bound |
|---|---|---|---|
| Private-call WER | 34.07 | 32.01 | 34.57 |
| PVSpeech CER | 4.77 | 4.63 | 5.07 |
| FLEURS CER | 12.89 | 12.13 | 13.19 |
The continuation began after the separate Rani-IndicConformer clean-speech pilot failed.
An earlier augmentation experiment improved synthetic noisy DEV error from roughly 28% to 22.3%. Real-call WER changed by +0.9 points, with interval [−0.9, +2.8]. The synthetic gain did not transfer to real calls.
What did beam search and the language model change?
On the parent checkpoint, TDT beam search improved DEV recognition over greedy decoding. Adding a language model further improved calls and PVSpeech:
| DEV panel | Greedy | Beam alone | Beam plus LM |
|---|---|---|---|
| Private-call WER | 30.62 | 29.38 | 28.71 |
| PVSpeech CER | 4.27 | 4.03 | 3.84 |
| FLEURS CER | 12.22 | 11.76 | 11.81 |
With the LM, FLEURS CER is slightly worse than with beam alone: +0.05 points, interval [−0.09, +0.20]. The decoder passed the predefined comparison against same-head greedy, which required better FLEURS DEV CER and no call or PVSpeech regression beyond 0.3 points.
The selected decoder uses mAES search, beam 8, a BPE 5-gram KenLM model, and LM weight 0.05. A secondary CTC experiment used a word 3-gram and weight 0.1. It passed against its own greedy control but produced weaker clean-speech results than TDT. TDT settings were tuned on the parent's DEV results and fixed for final Rani, including TEST.
The LM combined customer-call text with public Nepali text. Equality and containment checks screened out protected evaluation references and excluded call identities. Screened FLEURS train text made no final LM contribution; FLEURS train audio was used in acoustic training.
The LM was included in the first access-controlled private snapshot. It is not released because it contains customer-call text. Reproducing the beam-plus-LM scores requires that LM as well as the acoustic weights.
Which original release gates did the selected model fail?
Final Rani failed the original release gates in both decoding modes. The frozen rates below divide summed errors by summed reference units. PVSpeech is the reduced DEV panel used during development.
| Panel and metric | TDT greedy | TDT beam plus LM | Registered bar | Outcome |
|---|---|---|---|---|
| PVSpeech CER | 4.25 | 3.57 | <3.67 | Greedy fails; LM passes |
| FLEURS TEST CER | 10.7088 | 10.4319 | <7.43 | Both fail |
| NepTel v0.1.1 WER | 38.5763 | 37.5107 | <30 | Both fail |
| Private-call TEST WER | 28.3040 | 27.0704 | <47.7 | Both pass |
| Exact phone capture | 23/298 | 22/298 | ≥81/298 | Both fail |
The overall result is FAIL in both decoding modes. Both improve on the older call baseline but fail the FLEURS, old-reference NepTel and phone-capture gates.

What changed between the NepTel reference keys?
NepTel v0.1 used Chirp 2 drafts reviewed against the audio. Review corrected some errors and excluded unusable segments, but most usable drafts were accepted verbatim. The historical evaluation contained 75 segments and 2,375 reference words. Vendor processing had already applied noise reduction and personal-information muting to the audio.
The old references retained machine errors and shared label lineage with systems trained on related machine labels. The old benchmark also used a speaking-rate gate to exclude a severe repetition hallucination. The reference revision changed the evaluation without changing the acoustic model.
NepTel 0.2 contains 78 segments, 2,647 reference words and 31.5 minutes of 8 kHz phone audio. It combines 56 public customer-support segments with 22 voice-assistant-call segments spoken by the project owner. 13 earlier segments containing personal information were removed. The added speaker was not independently recruited.
A person listened to every current segment and corrected its reference against the audio. An internal Ampixa model provided the initial drafts and was excluded from the ranking.
WER divides substitutions, deletions and insertions by reference words, using totals across the corpus. The current reference convention treats equivalent English words in Latin and Devanagari scripts alike:
Rani has the lowest WER among ranked systems on this key. Each paired 95% interval against the listed competitors excludes zero.
| System | NepTel 0.2 WER |
|---|---|
| Rani, greedy, no LM | 33.51 |
| NepaliConformer offline | 39.71 |
| AI4Bharat IndicConformer | 47.60 |
The published paired 95% intervals favor Rani against every ranked comparator on this segment panel. Deltas are paired per-segment bootstrap means, which differ in the second decimal from differences between corpus rates. Both calculations use the same segment counts.
The older reference key ranks Rani below NepaliConformer offline. On the frozen 70-segment, 2,346-word v0.1.1 panel, greedy Rani's candidate-minus-offline WER difference is +4.90 points [ +2.39, +7.47 ]. With beam plus LM it is +3.84 [ +1.38, +6.42 ]. Rani leads on the current human-corrected key.
The reference text, scored subset, audio mix and script-equivalence convention differ between keys. Their scores are separate evaluations.
What did the private customer-call evaluations establish?
On the earlier private TEST panel, Rani scores 28.30 WER with greedy TDT and 27.07 with beam plus LM. The panel contains 3,451 utterance cuts from 128 calls and 34,776 reference words. Evaluation followed DEV selection and used fixed decoder settings.
The later, preregistered additional-call evaluation used a different held-out sample and Scribe utterance cuts. It scored 127 calls, skipped 1 empty-reference call, and contained 4,388 evaluation clips. The earlier 28.30 and later 30.6 WER use different utterance inventories and reference denominators.
| System | Later private-call WER |
|---|---|
| Rani, TDT greedy | 30.64 |
| AI4Bharat IndicConformer | 38.92 |
| NepaliConformer offline | 49.16 |
Rani outperforms each comparator but fails the absolute WER ceiling. The rounded WER values are 30.6, 38.9 and 49.2, respectively. The gate required Rani WER ≤30.0 and paired call-cluster interval upper bounds below zero against each comparator. Candidate-minus-comparator intervals are [−9.16, −7.44] and [−19.70, −17.39], respectively.
Call-identity and audio-hash checks excluded these Nepali customer-service calls from Rani training. Preparation also checked protected PCM and text identities. Speaker independence and inherited pretraining exposure were not established, and some historical source fingerprints were unavailable.
The references are machine-made Scribe labels without an independent human-transcription audit. Their label style overlaps the training convention. Training and evaluation calls came from the same organization and domain. The private references and hypotheses are unavailable for public inspection, and performance on unrelated organizations or call tasks was not tested.
How well does it recognize clean speech?
Clean-speech evaluation uses CER: character edits per 100 reference characters, including spaces after normalization. Rani has a lower PVSpeech point estimate with the LM, while AI4Bharat IndicConformer leads on FLEURS.
| System | PVSpeech, 93 sentences | FLEURS Nepali TEST, 726 sentences |
|---|---|---|
| Rani, greedy | 4.2 | 10.7 |
| Rani, beam plus LM | 3.6 | 10.4 |
| AI4Bharat IndicConformer | 9.8 | 7.4 |
PVSpeech helped select both the checkpoint and LM, including prompts later excluded for overlap. Half of the panel, the 50 sentences from its two adult speakers, is public on Hugging Face. Its four speaker clusters provide limited support for population inference.
AI4Bharat IndicConformer leads FLEURS at 7.4 CER. The fixed analysis uses 343 text-based grouping units because verified speaker identities were unavailable. Rani trails the base model on this read-speech test.

Exact entity capture
The entity test measured complete captures on 132 calls, split into 1,269 fixed 30-second chunks. The table gives complete captures divided by reference entities under the fixed capture procedure.
| Entity | Greedy hits / references | Beam plus LM hits / references |
|---|---|---|
| Amount | 84/152 | 85/152 |
| 0/123 | 0/123 | |
| Number with at least 3 digits | 67/331 | 69/331 |
| Phone number | 23/298 | 22/298 |
| Spelled code | 20/230 | 10/230 |
| Email domain | 1/124 | 1/124 |
Beam plus LM completely captures 22 of 298 phone numbers, below the required 81 of 298. Greedy captures 23. The LM lowers overall call WER but reduces exact phone-number and spelled-code capture. The normalization convention also limits literal digits, Latin spellings and email output.
How fast does it return text on a CPU?
Ampixa measured offline greedy transducer throughput on an Apple M2 Ultra at batch size 1. The test used 25 public NepTel clips totalling 622.72 s. “Times real time” divides audio duration by decode time. Peak process memory includes model loading and the runtime.

| System | Threads | Times real time | Decode s | Peak RSS MB |
|---|---|---|---|---|
| Rani | 1 | 26.81 | 23.23 | 1895.0 |
| NepaliConformer offline | 1 | 27.05 | 23.02 | 1868.0 |
| AI4Bharat IndicConformer | 1 | 21.08 | 29.54 | 1762.0 |
| Rani-IndicConformer | 1 | 20.79 | 29.95 | 1766.0 |
| Rani | 4 | 34.46 | 18.07 | 1983.0 |
On one thread of an Apple M2 Ultra, Rani runs at 26.81 times real time. AI4Bharat IndicConformer reaches 21.08, and NepaliConformer offline 27.05. The 4-thread Rani run used 35.9 CPU seconds, averaging 1.99 cores, with wall time 18.07 s.
First-text latency was measured after clip end, in seconds, with 5 trials per duration on 1 thread. It excludes audio capture time.
| System | 5 s clip | 10 s clip | 15 s clip | 30 s clip | 60 s clip |
|---|---|---|---|---|---|
| Rani | 0.20 | 0.35 | 0.48 | 1.21 | 3.52 |
| NepaliConformer offline | 0.20 | 0.33 | 0.48 | 1.20 | 3.52 |
| AI4Bharat IndicConformer | 0.21 | 0.41 | 0.62 | 1.57 | 4.32 |
| Rani-IndicConformer | 0.20 | 0.38 | 0.57 | 1.50 | 4.26 |
End-to-end demo measurements from Nepal included upload and ran on a free hosted CPU. A 10.3 s clip took a median 4.27 s; a 25.0 s clip took 7.20 s, with 4 trials each.
Rani and NepaliConformer offline used NeMo 3.0.0; the other models used NeMo 1.23.0rc0. Other work was running on the machine. Runtime versions, concurrent work, clip lengths and hosting overhead affect these comparisons. All timings are offline measurements; streaming partial-text latency was not measured.
The repository provides the offline, full-file inference implementation.
Rani-IndicConformer results
Rani-IndicConformer is a separate full-model fine-tune of AI4Bharat IndicConformer for clean read speech. Both pilots improved PVSpeech but failed the FLEURS non-regression gate. They used human-labelled OpenSLR, IndicVoices-R and FLEURS train, without call audio, pseudo labels or telephone augmentation. Both kept the base's native 256-piece Nepali tokenizer and language routing, a language-specific branch within the multilingual model.
The shared gate required PVSpeech CER to improve by at least 0.5 points, with FLEURS DEV CER regressing by no more than 0.3. Calls were evaluated but did not determine acceptance.
| Pilot | PVSpeech baseline → endpoint CER | FLEURS DEV baseline → endpoint CER | Verdict |
|---|---|---|---|
| Clean A | 9.79 → 8.01 | 7.49 → 7.86 | FLEURS fails |
| Clean B | 9.79 → 8.80 | 7.49 → 7.94 | FLEURS fails |
Clean A exceeded the FLEURS allowance by 0.07 points: regression was +0.37 against +0.30 allowed. Clean B regressed by +0.45, exceeding +0.3 despite greater FLEURS emphasis and a lower peak learning rate. Neither pilot proceeded to a long continuation.
A later fixed CPU evaluation of pilot B produced PVSpeech CER 3.89, FLEURS TEST CER 9.34, old-key NepTel WER 50.85, private-call TEST WER 50.49, and phone capture 43/298. The launch's 3.9 clean-speech result comes from that evaluation. The pilot DEV result was 8.80.
Checks of input and core source identities did not explain the PVSpeech difference between evaluations. The original gate remains failed. Native multilingual beam search was unsupported in the later evaluation, so it produced no comparable beam-plus-LM result.
How can someone reproduce the public result?
The repository and benchmark instructions provide public NepTel rescoring. Reproduction requires matching the dataset revision, canonical audio hashes, reference version, scorer revision, model identity and decoding settings. Historical results use their historical references and scorer. Set HYPOTHESES to the published Rani hypotheses before running:
cd benchmark
python fetch_audio.py neptel_audio
python ../eval/score_v02.py --hyp "$HYPOTHESES"
This re-scores published hypotheses without downloading a model. To evaluate another recognizer, transcribe all the same segments and pass segment/text JSON to the benchmark scorer. Keep segmentation fixed and include failed segments. NepTel references are for evaluation only.
After weight approval, full-file inference uses:
import nemo.collections.asr as nemo_asr
from omegaconf import OmegaConf, open_dict
model = nemo_asr.models.EncDecHybridRNNTCTCBPEModel.restore_from(
"rani-asr/rani/rani.nemo", map_location="cpu", strict=False
).eval()
with open_dict(model.cfg):
model.cfg.validation_ds = OmegaConf.create({"use_start_end_token": False})
model.change_decoding_strategy(model.cfg.decoding, decoder_type="rnnt")
result = model.transcribe(["audio_16k.wav"])
The loader uses a compatibility restore flag. Check restore diagnostics, the embedded tokenizer and the selected TDT head, and match waveform resampling and quantization conventions. The private evaluations used CPU inference; numerical equivalence on other runtimes is unverified.
Public hypotheses can be scored independently. Reproducing private-call results requires the private audio and references. Half of PVSpeech is public on Hugging Face. The LM and full training inputs are unavailable, limiting reproduction to the released components.
Where are the release files?
Weights, approval required · Code · Results · NepTel · Try the demo.
Code is MIT. Rani and Rani-IndicConformer weights use the Rani Model Licence. Distilling the model, using it or its outputs to create, train, fine-tune or evaluate another model, or distributing a derivative requires prominent Ampixa Labs attribution and passing that requirement onward. Existing NepaliConformer weights retain their separate terms. Acoustic weights require approval, benchmark outputs are public, and the LM is unavailable.
