Opinion

AI Leaderboards Play Deaf: Why Indian Benchmarks Never Make the Cut

High-quality benchmarks for the Global South already exist. The problem is governance: no one compels global leaderboards to include them, and market pressure only fixes outcomes, not the rules.

Font size

A new paper just dropped on arXiv (2608.18117) with a sharp thesis: India doesn't lack high-quality language benchmarks. It lacks a seat at the table. Datasets like IndicSUPERB, MILU, and LAHAJA already have enough academic credibility, yet ChatBot Arena, the Open LLM Leaderboard, and similar global rankings have never included them.

The paper puts it bluntly: "The barrier is institutional design... no governance mechanism compels them to do so." The obstacle isn't data; it's institutional design. Nothing forces leaderboards to act.

In other words, "we don't have the data" is a false premise. The real question is: who gets to decide which benchmarks count as "global," and why is that mechanism never opened to the Global South?


It's Not Missing Data, It's Missing Seats

To be clear: this isn't some Indian team publishing a grievance report about being overlooked. These three benchmarks are formally published, peer-reviewed, and actively maintained.

  • IndicSUPERB: covers 22 official Indian languages, with tasks spanning automatic speech recognition (ASR), speech translation, and language identification.
  • MILU: a multilingual understanding benchmark focused on Indian scripts, covering natural language inference and reading comprehension.
  • LAHAJA: a speech-language benchmark designed for low-resource scenarios, targeting real-world conditions: noise, accents, code-switching.

Taken together, these three already form a fairly complete evaluation infrastructure for the Indian region. The paper calls them "India-specific benchmarks" and groups them with IrokoBench (Africa) and AlGhafa (Arabic) as parallel cases.

Global leaderboards? Never included.

Why does this "not making it onto the stage" problem deserve a full paper? Because it punctures a long-standing excuse in the AI evaluation industry: "If African or South Asian models score poorly on the board, it's because the data isn't there." It is there. The system simply won't let it in.

Three Overlooked Names

Quick rundown of what each benchmark covers, so you can see these aren't afterthoughts.

IndicSUPERB

India's answer to SuperB: a multilingual speech benchmark covering 22 languages, including Hindi, Tamil, and Bengali. Tasks include ASR, speech translation, language identification, and speaker verification. Maintained by institutions like IIIT Hyderabad, it's already been cited in multiple top-tier conference papers.

MILU

Stands for Multi-task Indic Language Understanding. A multi-task Indian language understanding benchmark, essentially an Indian-flavored MMLU with a language set targeting the South Asian market. Covers text classification, QA, and natural language inference.

LAHAJA

Low-resource Audio-language Holistic Assessment of Joint Abilities. Focused on low-resource scenarios, with a defining trait: "real conditions," meaning noise, accents, and code-mixing. This is what everyday Indian users actually feed into a voice model.

Add all three together, and the paper argues you've got a coherent evaluation system for the Indian region. The global boards don't include any of it.

Leaderboard Governance Is Missing All Three Legs

The paper flags three structural gaps, and each is harder to fix than adding a dataset.

1. Independent Governance

ChatBot Arena is the hottest general-purpose leaderboard, run by LMSYS, a group of researchers from UC Berkeley, Stanford, and UT Austin. Hugging Face's Open LLM Leaderboard is community-run. OpenAI, Google, and Anthropic's internal evaluations aren't public at all.

Where does this break down? The decision of "which benchmarks to include" is made unilaterally by the operators. No external committee, no mandatory consultation, no public comment period.

The paper's recommendation: establish an independent "benchmark inclusion committee" with members from academia, regional representatives, and civil society groups, structured something like an IEEE standards committee.

2. Conflict-of-Interest Policy

LMSYS takes sponsorship from cloud vendors. Hugging Face receives funding from multiple model companies. OpenAI runs its own evaluations, and its own models are always at the top.

Not a single global leaderboard has a public conflict-of-interest policy. No requirement to disclose funding sources, no recusal clauses, no third-party audit.

This is why the "selective neglect" of Indian benchmarks looks suspicious, not because of conspiracy, but because of an institutional vacuum. Operators can reject a benchmark for any reason (workload, compute costs, business relationships) and they don't owe anyone an explanation.

3. Metric Evolution

Once a benchmark hits the board, it gets gamed until it stops working. MMLU has already gone this route. HumanEval is nearly saturated by the GPT-4 family.

But global leaderboards generally lack a formal retirement or refresh process. Metric evolution is manual, ad hoc, and slow. The paper argues evaluation systems need version control, automated drift detection, and periodic redesign.

All three of these (independent governance, conflict-of-interest policy, metric evolution) are not technical problems. They're governance problems.

Market Pressure Fixes Half of It

The paper actually concedes a point here: commercial competition will naturally surface good models, which is something the market can handle.

Aravind Srinivas won't post on ChatBot Arena, but Perplexity will actively run Indian language cases because losing the Indian market is not an option. OpenAI won't proactively test IndicSUPERB, but Google will optimize multilingual metrics to pull Gemini's share in India.

But "making it onto the board" is something the market can't solve.

A model with strong scores that isn't on the board effectively "doesn't exist" in academic citations, policy reports, or enterprise procurement lists. This signal gap is an institutional problem, not a technical one. Commercial pressure can close "performance gaps," but not "perceptual visibility" gaps.

My read: technical problems get marketized; governance problems don't. The former runs on competition; the latter runs on rules.

Africa and Arabia Are in the Same Boat

India isn't unique. The paper groups IrokoBench (Africa) and AlGhafa (Arabic) as parallel cases.

  • IrokoBench: covers around 20 African languages, including Yoruba, Hausa, and Swahili, all low-resource.
  • AlGhafa: an Arabic NLP benchmark targeting Modern Standard Arabic and dialectal variants.

Both are formally published and academically recognized. Global boards don't include them either.

This isn't "discrimination against India." It's a systematic geographic bias.

When a leaderboard can only tell you "GPT-5 beats Claude by 3 points on English," it's fundamentally optimizing a monolingual, monocultural, single-market view of evaluation. The rest of the world, whose combined population vastly exceeds native English speakers, gets bucketed as "long tail" by default.

Uneven dataset distribution is fine on its own. The "global" label on these boards is fake. That's the problem.

Next Step: Treat Leaderboards Like Code Audits

What can actually be done?

The paper's direction: treat evaluation systems as public infrastructure that requires external auditing. Three concrete moves:

  1. Mandate disclosure of inclusion/exclusion reasoning. When an operator rejects a benchmark, they must state why publicly (compute cost, relevance, scale).
  2. Establish an independent committee: academia + regional representatives + civil society, modeled on journal editorial boards.
  3. Build version control and drift detection. Once a benchmark goes live, audit it periodically; if it's been gamed into the ground, trigger a redesign.

This sounds like boilerplate from a policy paper, but the underlying claim is: evaluation isn't a neutral tool; it's a product with embedded values. It should be regulated, not worshipped.

What does this mean for practitioners? Short term: keep watching GPT-5 vs Claude vs Gemini on ChatBot Arena. It's still the best proxy for English-language tasks. Long term: stop treating "global leaderboards" as gospel. The world they cover is much smaller than you think.

If an AI product wants to break into the Indian market, it won't ride on MMLU scores. It'll ride on IndicSUPERB plus real user feedback. Leaderboards are marketing. Evaluation is engineering.


Reference: arxiv.org/abs/2608.18117

Source: https://arxiv.org/abs/2608.18117