Products

Dayjob Tax: Can AI Agents Handle Tax Work?

Surges tested an AI Agent across 50 professional tax tasks, focusing on missing data, calculation traceability, citation quality, and production permissions rather than aggregate accuracy alone.

Font size

Tax work is a strong test for an AI Agent.

The questions appear straightforward, and the results can be delivered as text. In practice, however, the work exposes whether source materials are complete, calculations are correct, conclusions can be traced back to authoritative guidance, and outputs can be independently reviewed.

This adds a layer beyond ordinary question answering: the answer is not the endpoint; it is the start of a delivery process with defined accountability.

Dayjob: Tax, published by Surges, tests that capability through 50 professional tax tasks set in a specialized environment with explicit compliance requirements.

50 Tax Tasks Go Beyond a Q&A Test

Dayjob: Tax does not ask a model to explain tax concepts or answer a handful of common questions.

Instead, it sets 50 professional tasks that typically require a sequence of actions:

  • Finding and reading business records;
  • Identifying applicable rules and calculation methods;
  • Calling calculation or lookup tools;
  • Organizing the results into a deliverable document;
  • Citing supporting sources for human review.

In a conventional Q&A test, a plausible response is enough. Professional tasks are more demanding. If one required input is missing or the wrong rule is cited, the result may be undeliverable even when the number looks reasonable.

This is the practical distinction between an Agent and a chatbot. An Agent must generate an answer, but it also needs to use tools, maintain context, and handle failures midway through a workflow.

Dayjob: Tax places that distinction in a tax setting, where professional judgment must be paired with explainable results. A tax return with no traceable basis carries a different risk from one supported by complete references but still requiring corrections.

For that reason, overall accuracy is not enough. The evaluation also matters because of the tasks it covers, the tools the Agent can use, and whether its deliverables contain the information needed for review.

Weaknesses May Hide in the Gaps Between Steps

A correct answer to a single question does not guarantee that an Agent can complete an end-to-end task reliably.

When handling tax work, failures often occur in the middle of the workflow.

1. Does It Keep Calculating When Data Is Missing?

In real work, attachments rarely arrive with every field populated and neatly labeled.

Income may appear in an email, deduction records may be in a PDF, and a tax-rate adjustment may be documented in a separate policy file. If an Agent cannot access those materials, it should report the gap rather than fill it with a plausible-looking number.

The most dangerous failure is not an explicit error. It is a silent assumption.

2. Can It Keep Multi-Step Calculations Consistent?

Tax calculations usually form a chain: subtract allowable deductions from income, apply the relevant rules to determine the taxable amount, and then account for withholding, credits, or additional charges.

Using the wrong method at any stage can distort the final result. The larger problem is that an Agent may show only the final number without exposing the intermediate values.

People cannot review a number produced by a black box. At minimum, the Agent should preserve key intermediate values, formulas, and data sources.

3. Do Citations Support the Conclusion?

Including a few links does not amount to providing supporting evidence.

An Agent needs to identify the specific rule it relied on, explain the conditions under which it applies, and show how it affected the calculation. A citation that does not support the conclusion can be harder to detect than no citation at all.

In a compliance-sensitive setting, citations are not decoration. They form part of the audit trail.

4. Is There a Clear Path to Review?

A professional deliverable should not leave users to infer the final step.

A more reliable pattern is to output the conclusion, calculation process, original supporting sources, and items requiring confirmation. People can then identify where judgment is needed without rebuilding the entire analysis from scratch.

This is where Dayjob: Tax resembles enterprise work more closely than a conventional benchmark: it tests deliverability, not simply whether the Agent got the answer right.

Average Scores Are Insufficient in High-Compliance Work

An average across 50 tasks is useful for comparison, but it cannot independently prove that an Agent is ready for production.

Suppose an Agent answers 45 tasks correctly, producing a 90% average. Five tasks still fail. If three involve an incorrect filing, omitted materials, or an untraceable conclusion, the consequences remain significant.

For high-compliance use cases, I would examine four measures.

First, performance on the weakest tasks. The minimum score can reveal the input conditions under which the Agent breaks down, which may be more useful than its overall ranking.

Second, the rate of severe errors. Errors should be classified. Incorrect conclusions, faulty citations, calculation errors, and omitted confirmation items should not be collapsed into a single metric.

Third, traceability. Every conclusion should lead back to its input data, calculation steps, and original supporting sources. Without that path, even a high average accuracy makes accountability difficult.

Fourth, how failures present. A good failure states, “The 2025 deduction receipt is missing, so this amount cannot be confirmed.” A bad failure continues calculating and returns an answer with unjustified confidence.

Together, these measures determine not whether an Agent is intelligent, but whether an organization trusts it with the next assignment.

Production systems should also impose narrower permission boundaries. Access to source documents does not imply authority to submit filings. The ability to calculate does not confer authority to choose the final filing position. Having an Agent produce a result for confirmation by system rules or qualified professionals is generally safer than granting open-ended permissions.

What 73 Points and 46 Comments on HN Mean

The project reached the Hacker News front page, earning 73 points and 46 comments.

That is not the response of a viral launch, but it shows that developers were willing to discuss the project seriously.

The 73 points gave the project visibility, while the 46 comments moved the conversation beyond a simple leaderboard share. Developers asked whether the tasks were realistic, whether the scoring was credible, how errors were classified, and which restrictions could not be bypassed during enterprise deployment.

The original evaluation is available here: Dayjob: Tax

The discussion is available here: Hacker News discussion

Developers were not looking for another model leaderboard. They were asking a more practical question: can an Agent own a professional task with clear accountability inside a real enterprise workflow?

That is the gap that professional Agent evaluations need to address next. They should publish more failure examples, document the data and tools required for each task, and define deliverables that can be independently reviewed. Otherwise, an attractive overall score can still conceal the most costly mistakes.

Editorial Take

The most important signal from Dayjob: Tax is not the score achieved across 50 tasks. It is the shift in professional evaluations from asking whether an answer is correct to asking whether the workflow can produce a defensible deliverable. Unlike general question-answering benchmarks, tax work depends on complete source materials, calculation chains, supporting citations, and human review, making accuracy figures insufficient on their own.

My view is that the next generation of genuinely useful AI Agents will not have unlimited permissions. They will stop at risk points. What enterprises need may not be an Agent that acts more boldly, but a workflow with narrower permissions and stricter validation.

The evaluation’s details still remain uncertain. If task inputs are overly organized or failure severity is not strictly distinguished, traceability may exist only as a presentation feature. The available evidence confirms that Surges built an evaluation with 50 professional tasks. Whether it predicts the risks of real filings will depend on subsequent case studies.

Source: https://surgehq.ai/blog/dayjob-tax