Hugging Face adds Hindi and Indian English to its speech recognition benchmarks
Models that score well on the Open ASR Leaderboard get adopted. Capabilities the leaderboard does not measure tend not to improve. Recent work has focused on making the evaluation metrics more trustworthy.
- Held-out private splits.
- Benchmark-fitting analysis to quantify how much models are reproducing reference transcripts rather than transcribing solely on the audio.
- Closing the gaps in normalisers to ensure correct predictions and variants are not penalised.
All of that makes one number (WER) harder to game. It is still one number. A long line of work has shown that ASR error rates are not evenly distributed across the people using them. Racial disparities in automated speech recognition found commercial systems roughly twice as bad for Black speakers as for white speakers, and Quantifying Bias in Automatic Speech Recognition found further differences by gender, age and accent. None of that is visible on a leaderboard, and not because the leaderboard is hiding it. The test sets it runs on record what was said and almost nothing about who said it.
To address this gap, we introduce two evaluation sets to the Open ASR Leaderboard: Monsoon en-IN and Monsoon hi-IN. Hindi, spoken by more than half a billion people, is the first Indic language on a multilingual tab that currently covers only European languages. Each set is released as a public split, available for self-scoring, and a private split withheld to limit benchmark-specific optimisation. The four splits are speaker-disjoint, comprising 4,888 speakers, with 12 speaker attributes recorded for each.
Design of the collection
A test set can only expose a failure mode it varies along. Most benchmarks are built from whatever audio was readily available. Monsoon was built to vary along nine axes: geography, age, gender, vocabulary, devices, acoustic environments, speech type, speech rate, and the existence of multiple valid transcripts for the same audio. Each is a way an aggregate WER can be right on average and wrong for a particular population.
The collection method follows from that.
- Geography comes from recruiting across hundreds of districts rather than recording longer sessions in fewer places.
- Devices and acoustic conditions come from contributors using their own handsets and connections, indoors and out, rather than supplied hardware in a quiet room.
- Vocabulary, speech type and speech rate come from the prompts: everyday topics that push contributors toward opinion, disagreement, narration and recall, which is where named entities, numbers and unrehearsed phrasing appear.
- Age and gender are recorded per speaker and verified.
- Multiple valid transcripts is a property of the reference rather than the audio, and it is the subject of a later section.
Dataset composition
Four splits, two languages, collected through one pipeline.
The data is sourced from unscripted dual-channel spontaneous conversations, with clips segmented from a single channel so that each clip carries one speaker. Along with the fields reported in the table, each clip also records occupation, education, marital status, income band, handset brand, current city and years in the current district.
Five clips from the public Indian English split, with the metadata each one carries:
29-year-old woman, West Tripura, Tripura. Student, samsung SM-G781B.
32-year-old woman, Satna, Madhya Pradesh. Unemployed, samsung SM-E146B.
22-year-old man, Rohtas, Bihar. Student, motorola moto g54 5G.
27-year-old woman, Warangal, Telangana. Unemployed, vivo V2247.
57-year-old man, Puducherry. Private job, Xiaomi M2006C3LI.
The English sets use standard string references, where the leaderboard’s normaliser collapses most spelling variation. Hindi has far more of it, and no normaliser can resolve it, because the variants are not a fixed mapping between two conventions. The Hindi sets therefore ship a lattice: for each span of the transcript, a list of the spellings that are accepted as correct.
Speaker coverage
Monsoon is small measured in hours and large measured in speakers. That is the design, and it is where most of the value sits.
Speaker concentration and diversity beyond the fields above.
Three properties follow, and each is a claim about variance rather than volume.
No voice carries the score: The ten largest contributors account for between 2.8% and 6.8% of total duration, and more than half of all speakers appear exactly once. A result on Monsoon is an average over hundreds of distinct voices, not a small number of talkers recorded at length. Test sets of comparable duration are usually constructed the other way.
No region or handset carries it either: The Indian English public set draws on 428 native districts across 30 states and union territories; the Hindi sets, being a Hindi-belt language, concentrate more tightly but still span 202 and 295 districts. Recordings come from 315 to 582 distinct device models, with no single model exceeding 2.1% of segments in any subset. Corpora collected on standardised hardware overfit to one microphone response; this one cannot.
Indian English here is not one accent: This is English as it is spoken across the country, not the English of one region. All six zones are represented: in the public set, 35% of segments are contributed by southern speakers, 18% from the East, 18% from Central
Source Read original →
