Yesterday the Hacker News front page featured a Show HN: Strands Harness post that generated a fair amount of discussion. I clicked through and found that the AWS Strands team had packaged two things — running Agent tasks and scoring the results — into a standalone tool called Strands Harness.
The name sounds unglamorous, but what it does is specific: it records the full multi-step execution of an Agent, lets you replay a failure from the beginning, and scores the run after it finishes. The Agent community has been calling for proper observability for a while, and AWS has now delivered tooling on that front.
What Strands Harness Is
One line: the official debugger and evaluator for Strands Agents.
Some context first. Strands Agents is an open-source Agent SDK AWS launched in 2025, positioned similarly to LangGraph or CrewAI — it lets developers assemble Agents in Python that can call tools and run multi-step reasoning. Harness doesn't replace the SDK; it complements it. The SDK handles "making the Agent run," while Harness handles "checking whether it ran correctly."
The official blog describes it as a "multi-agent task execution and evaluation harness." In plain terms: you write a task description, Harness automatically orchestrates a group of Agents to execute it, logs every step along the way, and scores the result against rules you define.
Why You Need Replay and Evaluation
I once built a multi-step Agent in LangGraph whose job was to pull data from Notion, format it into a report, and post it to Slack. I ran it 50 times, and 7 of those runs crashed on the third tool call. The logs just told me "tool call failed."
As for why it failed — what prompt the LLM saw, what parameters the previous step passed — I had to add print statements manually to reproduce.
This pain point is known as the Agent black box: a single LLM call is easy to debug, but multi-step Agents running long chains of tasks make a single failure nearly impossible to diagnose from scratch.
That's what Harness aims to fix. The official blog gives a real scenario: have an Agent fetch weather for 5 cities and generate travel recommendations. Multi-step tool calls, any one of which could fail. Did the API return null? Did the LLM break parsing the JSON? Was a step truncated because of token limits? Traditional trace tools (LangSmith, Langfuse) can show you "the input and output of each step," but not the full task context. Harness runs the complete task: it weaves multiple Agent trajectories into a replayable story.
Architecture Breakdown: Pulled From the Blog
I dug through the official blog (strandsagents.com/blog/introducing-strands-harness/), and Harness has three core components:
1. Task Runner
Takes a task description defined in YAML or JSON, containing Agent configuration, tool list, and evaluation criteria. The Runner schedules multiple Agent instances to run in parallel or sequence, recording the input, output, latency, and token count for every step.
2. Replay Harness
This is the biggest differentiator from existing trace tools. LangSmith and Langfuse offer post-hoc trace views — you can see what happened at each step, but you can't "re-run that step with a different prompt and see what happens."
Harness's replay is deterministic: you can click any step in the UI, change its input parameters, re-run just that step, and see how downstream steps react.
The official blog emphasizes that this design is "end-to-end replay," not single-step tracing. For debugging multi-step Agents, this means you can quickly pinpoint "which step's state is poisoning downstream," instead of guessing from scratch.
3. Evaluator
After a task run completes, Harness calls an LLM judge (Claude by default) to score the output against your rubric. The rubric can be structured (e.g., "the final answer must include city names") or natural language ("evaluate the readability of the report"). Scores get written into the replay record, making it easy to compare different prompts or models side by side.
Here's a real code example from the README, simplified:
from strands_harness import Harness, Task
# Define a multi-Agent task
task = Task(
name="weather-report",
agents=["planner", "weather-fetcher", "writer"],
tools=["get_weather", "send_to_slack"],
eval_rubric="Final output must be structured JSON containing a cities array and a summary field",
)
# Run the task; Harness automatically records the full trajectory
result = Harness(task).run(replay=True)
print(result.score) # 0.85
print(result.replay_url) # https://harness.example.com/replay/abc123
(This is a simplified illustration; see the official blog for the full API.)
How It Compares to Existing Tools
- LangSmith — LLM trace and monitoring. No Agent task replay (single-step trace only). Basic evaluation built in. Vendor-locked to LangChain.
- Langfuse — Open-source LLM observability. No Agent task replay (single-step trace only). Evaluation via plugins. No vendor lock-in.
- Helicone — LLM API proxy and logging. No replay. No built-in evaluation. No vendor lock-in.
- Strands Harness — Agent task execution + replay + evaluation. End-to-end replay. Built-in LLM judge. Vendor-locked to AWS/Strands.
The differentiation jumps out immediately: Harness isn't a trace tool with extra features; it's a complete Agent testing platform. Running tasks, replaying them, scoring them — all in one tool.
But the lock-in is real — Harness is tightly coupled to the Strands Agents SDK. If your Agent is built with LangGraph or AutoGen, you basically can't use it (unless you write an adapter). That's AWS's play: bind the entire Agent development workflow to the Strands ecosystem and keep developers from drifting back to LangChain's toolchain.
Editorial Take
Why does this matter? What I see is the Agent ecosystem shifting from "can run" to "runs reliably."
During the 2024 Agent framework wars, everyone competed on "who can make an Agent run" — LangGraph, AutoGen, and CrewAI all battled on ease of use, tool calling, and multi-Agent collaboration. But in production, the real pain isn't "the Agent doesn't know how to call tools" — it's "the Agent called the wrong tool and I don't know why." The observability space has been dominated by trace tools like LangSmith and Langfuse, but trace tooling addresses the LLM call layer. State contamination, backtracking, and evaluation in multi-step Agents are problems traditional trace tools can't handle.
Harness's significance is that it tears open the "task-level Agent observability" category. End-to-end replay plus a built-in LLM judge is something trace tools either haven't figured out or don't want to build.
But AWS's play behind this is obvious too — locking down the Strands ecosystem. LangChain never owned trace and evaluation well, and LangSmith filled that gap. AWS is now staking out the Harness layer early, so teams using the Strands SDK naturally adopt Harness for evaluation, and switching to LangGraph later means rebuilding the evaluation stack. This is a platform strategy, not a product strategy.
My prediction: 2-3 leading Agent companies (open-source or closed) will launch similar "task-level evaluation platforms" in the coming months. If LangChain doesn't respond, LangSmith will get squeezed into the "just a trace tool" niche — and that's bad news for them.
What I'm unsure about: Harness is still very new, and the docs don't show any large-scale production case studies. "End-to-end replay" sounds great, but in genuinely complex multi-Agent systems, will state management become unmanageable? That'll take six months of community feedback to answer.
Source: Strands Harness: Multi-Agent Execution and Evaluation - Strands Agents Blog