Mercor released a study this week called the Labor Index, and the accompanying Benchmark Map puts a number on something the field has long suspected: roughly 90% of US working hours happen in domains that mainstream AI benchmarks barely touch.
The remaining 10%? Software engineering, math, code completion. SWE-bench, MATH, and HumanEval essentially own that slice.
Caregiving, food service, sales, customer support, administration—these categories employ a large share of the US workforce, yet you won't find a single public benchmark on HuggingFace that targets them specifically.
What Mercor Actually Built
The methodology is straightforward. Mercor took the US Department of Labor's O*NET (the Occupational Information Network, where each occupation maps to a detailed task list) as the base layer, categorized every work task in it, and then checked each one against existing AI benchmark problems—is this task measured by any benchmark?
The result is a coverage map. Its significance: it places "what AI can do (benchmarks)" and "what people do for money (O*NET)" on the same coordinate system.
No one had done this before. Researchers in the two camps ran on separate tracks: benchmark builders watched public leaderboards; labor economists watched BLS statistics. The datasets never lined up.
Mercor is an AI hiring platform—more on their motivation later. But the "first map" position is theirs to claim.
The Numbers: 90% / 10%
Mercor's coverage breakdown:
- Programming tasks (SWE-bench, HumanEval, MBPP, etc.): well covered
- Math reasoning (MATH, GSM8K, AIME series): moderate coverage
- Real-world tasks—cross-role communication, multi-step procedures, ill-defined problems: near zero coverage
Almost every mainstream benchmark today is essentially measuring "things AI is already good at."
Mercor's headline finding in one sentence: current benchmark coverage is fundamentally misaligned with the actual US labor market.
The Jobs the Field Ignores
What gets left out? Mercor groups it into a few categories:
- Care and health support: caregiving, hospital navigation, therapy assistance
- Food service: order taking, inventory, handling surprises
- Sales: B2B follow-up, negotiation, relationship management
- Customer support: first-line reps, returns, de-escalation
- Admin and operations: data entry, cross-department coordination
Why don't benchmarks measure these? Three layers of reason:
- Hard to auto-grade—unlike code with test cases, "was that support response good?" requires human judgment.
- Heavy on common sense and interpersonal interaction—multi-turn dialogue, emotion reading, parsing fuzzy intent.
- No clear right answer—"Was this sales pitch good?" "Did this menu recommendation land?" The scoring criteria are inherently subjective.
Benchmarks naturally favor quantifiable tasks with standard answers that can be batch-evaluated. Math problems and code problems thrive. Caregiver scripts, restaurant scheduling, inventory optimization—work that accounts for a huge chunk of GDP—the AI labs collectively pretend it doesn't exist.
This Means Vendors Are Optimizing the Wrong Thing
When OpenAI's new model hits a perfect AIME score, or Anthropic reaches near-human-expert performance on SWE-bench Verified, what are they actually optimizing for?
Answer: a metric that is severely disconnected from real economic value.
Analogies help. Back in 2012, ImageNet sent every vision company scrambling, only for the field to later discover a vast gulf between "classify 1,000 object categories" and "an autonomous car that spots a kid by the road." High benchmark scores don't translate to high real-world scores.
That same gulf is reappearing in the LLM era. This time it's more insidious, because programming and math do have genuine economic value—labs can defend leaderboard-chasing with "AI now writes half your code." But what determines AI's long-term GDP impact is those "hard to measure" industries—and not a single decent benchmark exists for them.
The result is a severe misallocation of capital and engineering talent. Every top lab is fighting over the same 10% of the cake. The remaining 90% doesn't get a glance.
Mercor's Motivation and the Dataset's Limits
Mercor isn't a neutral academic institution. It's an AI hiring platform whose business model is helping companies evaluate candidates with AI. The timing of this report—pushing the idea that "benchmarks are broken, so you need us"—clearly previews their own workforce assessment product.
That motivation has to be acknowledged.
What are the study's actual limits?
- Source data is O*NET, which is fundamentally a job description database, not actual hours-worked data.
- Country coverage is the US only—the labor structures of China, India, and Europe look very different.
- Industry granularity stops at "occupation category," not specific roles.
- The definition of benchmark coverage is itself a judgment call—does "customer support dialogue" count as tested if an LLM-as-judge benchmark touches it? Mercor defined that itself.
Even so, the map is useful. It quantifies something nobody had before: the AI benchmark world and the real labor market are two separate universes.
Further Thoughts
This goes deeper than it appears.
It explains why GPT-5 and Claude 4.5 can score 99% on SAT math, yet booking a 6 AM connecting flight tomorrow still requires them to confirm with you five times before committing—because "book a flight" simply doesn't exist in benchmark-land.
How is this different from before? This disconnect is structural. Earlier benchmark-vs-reality gaps were typically difficulty gaps (Deep Blue beats Kasparov but loses to nobody among amateurs). Today's problem isn't difficulty—it's direction. Benchmarks measure what you're already good at, while the work that actually drives GDP impact isn't even being measured.
My prediction: over the next 2–3 years, benchmark inflation will continue—SWE-bench is about to saturate, harder benchmarks will appear, then saturate too. At the same time, "real economic value" benchmarks will slowly emerge—possibly customer support dialogue scoring, possibly caregiver workflow simulation, possibly sales close-rate prediction. Once these new benchmarks take hold, the AI vendor leaderboards will reshuffle.
What I'm less sure about: whether "real-work benchmarks" can survive in the academic community. They inherently run into data privacy, human-rater consistency, and standardization problems. They may end up as internal corporate tests with no public leaderboard—which isn't necessarily good for the ecosystem.
But the direction is clear: the second half of the leaderboard era isn't about harder problems—it's about problems that have never been leaderboarded at all.