Frameworks

FinSkillBench Targets the Gap Nobody Wants to Talk About: Auditable Investment Agents

General benchmarks test whether an agent can talk. FinSkillBench tests whether it can actually close the books—across portfolio construction, risk management, and fundamental analysis.

Font size

Hand a financial analyst GAIA or AgentBench and watch what happens. They'll point out the obvious within minutes: these benchmarks measure whether an agent can call tools, not whether the financial calculations are correct.

Then comes the harder problem: auditability. When an agent gets a historical P/E wrong, nobody notices. When it gets 2008 market exposure wrong, regulators catch it at reconciliation.

FinSkillBench (arXiv:2608.18099v1) draws a line in that gap.

What Existing Financial Benchmarks Miss

GAIA covers general reasoning and tool use. AgentBench spans web browsing, database operations, OS tasks. Neither validates financial calculation accuracy, and neither cares whether the output can be audited.

Several finance-specific concerns get overlooked:

  • Data timeliness: Investment work demands point-in-time data. A question about 2008 can't reference a 10-K published in 2010.
  • Auditable intermediate steps: When an agent outputs "portfolio volatility 12%," it should also show which return series it used, how the covariance matrix was estimated, and what decay factor was applied—otherwise compliance can't reproduce the result.

FinSkillBench explicitly classifies investment management as a high-stakes domain and requires agents to complete a four-step chain:

  1. Retrieve point-in-time data
  2. Assemble correct calculation inputs
  3. Apply domain-appropriate methods
  4. Output auditable, structured reports

Miss any step and the result can't be trusted.

Three Test Zones: Portfolio, Risk, Fundamentals

The paper maps these three domains directly to real investment workflows.

Portfolio construction: Given an investment universe and constraints, produce an optimal allocation. Common failure modes: wrong benchmark returns, miscalibrated risk aversion, ignored turnover costs.

Risk management: Compute VaR, run stress tests, estimate factor exposures. Common failure modes: using future data in historical backtests, wrong covariance windows, linear approximations of nonlinear risk.

Fundamental analysis: Break down financial statements, compute valuation multiples. Common failure modes: wrong accounting standards, treating quarterly figures as annual, peer companies in the wrong industry.

The common thread across all three: agents don't just need to get the number right—they need to document how they got there.

The Hard Part Isn't "Being Right"—It's Surviving an Audit

General QA benchmarks reward correct final answers. Financial scenarios impose a stricter bar: audit requirements.

Two constraints drag agents back into engineering reality:

  • Point-in-time data retrieval: The system needs to know exactly which date you're asking about. In practice, analysts search by "data availability date"—an annual report released March 31 isn't usable in a portfolio model until April 1. An agent that doesn't distinguish "event date" from "available date" introduces look-ahead bias.
  • Auditable structured output: Results have to be programmatically reproducible. "Volatility is roughly 12%" isn't enough. You need something like {"portfolio_id": "X", "method": "historical_var_1y_99", "value": 0.12, "inputs": [...], "timestamp": "..."}—a report with fields that can be checked.

These two constraints are what separate FinSkillBench from general-purpose benchmarks. The paper's abstract states the evaluation goal plainly: "whether language model agents can effectively use financial domain skills"—not linguistic fluency.

Can Agents Actually Deliver? On Evaluation Methodology

FinSkillBench's scoring logic checks two things:

  • Whether the result is correct: Does the final number fall within an acceptable range?
  • Whether the process is correct: Are intermediate steps reasonable—right data source, appropriate method, point-in-time consistency maintained?

This design reflects a deliberate tradeoff. Results-only scoring lets agents game the system by guessing a few numbers right. Process-only scoring ignores real business value. Combining both mirrors real-world investment workflows' double-check pattern: review the logic before the trade, verify the numbers after settlement.

The evaluation target is explicit: effective use of financial domain skills. That's a different goal than measuring language fluency (as some LLM-as-judge frameworks do)—it's closer to traditional financial software validation: unit tests, integration tests, manual audits.

Does the Vertical Benchmark Path Work?

FinSkillBench isn't alone. Vertical benchmarks are sprouting across healthcare, legal, and finance.

The underlying bet: broad benchmarks like MMLU are losing signal. Three reasons drive this:

  • Models are converging on general tasks: The gap between an MMLU score of 80 and 85 says less and less about actual capability.
  • Industry scenarios need specialized knowledge: Financial calculations, legal provisions, medical diagnostics—general benchmarks don't reach them.
  • Auditability is an engineering necessity: General benchmarks don't validate output structure; vertical use cases demand it.

Will "small and specialized" benchmarks carry more signal than "big and broad" ones? My read: yes—provided the designers actually understand the domain. FinSkillBench's designers do: they know investment management is high-stakes, they know the four-step chain must hold, and they know audit requirements are non-negotiable.

Back to the original question: who fills the investment-agent evaluation gap? FinSkillBench is a start, not a finish. When a downstream developer runs this benchmark against their fine-tuned model and finds risk calculation accuracy stuck at 60%—that's where vertical benchmarks earn their keep. They don't test whether you can talk. They test whether you can do the work.

Source: arXiv:2608.18099 - FinSkillBench

Source: https://arxiv.org/abs/2608.18099