Eval Harnesses for Web-Grounded Agent Outputs

Standard eval harnesses were built for a world where the right answer sits still. Web-grounded agent outputs do not sit still: the correctness criteria, the ground truth, and the expected sequence of steps all shift as the pages underneath them change. A classic harness runs on a fixed golden dataset, scores with exact-match or near-match logic, and reports a pass or fail against a threshold. That design works fine for closed-domain tasks, where the correct answer today is the correct answer next year.

Web-grounded agents break every one of those assumptions at once. The content an agent retrieves on Monday may differ from what it retrieves on Friday. The tool calls an agent makes depend on the state of the page it hits, so two runs against the same prompt can produce two legitimately different (and both correct) trajectories. An answer graded as correct this morning can become wrong by afternoon, through no fault of the model.

Early agent benchmarks tried to handle this with exact trajectory matching: score a run by how closely it mirrors a ground-truth demonstration, step for step. That method holds up in offline, static settings. It falls apart in a dynamic web environment, where there is no single correct trajectory because the web itself has more than one valid path through it at any given moment.

The discipline around harness engineering has caught up to some of this. Harnesses now cover workflow design, evaluation, permission controls, and persistent state management, a far wider scope than the prompt templates that defined early agent frameworks. But broader scope is not the same as a fix for the core mismatch. The architecture is what needs fixing: what "correctness" is assumed to mean, regardless of which tools get bolted onto the harness. Any real fix has to start by redefining that term before a single line of scoring code changes.

The five evaluation dimensions under web dynamism

Agent output quality is commonly organized across five dimensions: Correctness, Groundedness, Safety, Trajectory, and Performance. Correctness covers metrics like ExactMatch, TaskCompletion, GEval, and GoalAccuracy. Groundedness covers Faithfulness, SpecGrounding, ContextPrecision, and AnswerRelevancy. Safety covers PII, Toxicity, PromptInjection, and Hallucination. Trajectory covers PlanAdherence, StepEfficiency, and ToolCorrectness. Performance covers Latency, TokenCost, and CostEfficiency. Each one takes a different kind of hit once the agent's inputs come from a live crawl instead of a fixed dataset.

Correctness suffers first and most visibly. ExactMatch assumes the golden answer stays true. When the source page updates and the golden doesn't, a score of zero can mean the agent is wrong, or it can mean the golden is stale. The harness has no way to tell the two apart without extra machinery.

Groundedness is the hardest of the five to measure well under web dynamism. A system can score high on a holistic faithfulness rubric while audits turn up hallucinations in a meaningful share of the responses reviewed, because a holistic score averages support across the whole answer instead of checking each claim inside it. An answer that leans on one supporting chunk while quietly ignoring a contradicting chunk will sail through a holistic check and fail the moment someone decomposes it claim by claim.

Safety takes on a new attack surface the moment content comes from the open web: prompt injection embedded in a crawled page is not something a static golden dataset ever has to contend with. Safety metrics need to be scored on their own, never folded into a composite number where a high correctness score can paper over an injection risk.

Performance degrades in a way that's easy to misread. Crawl latency, JavaScript rendering overhead, and nondeterministic extraction all add variance to latency and token cost. A harness calibrated on static inputs will blame that variance on the model, when the real source is the web-fetching layer sitting in front of it.

Trajectory, covered in depth further down, breaks in its own distinct way: the "right" path through the web is rarely singular, so judging a run against one fixed expected path misses valid alternate routes and lets genuinely broken ones through. Taken together, the five dimensions show that web dynamism does not single out one weak point in a harness. It degrades correctness, groundedness, safety, performance, and trajectory simultaneously, each for a different structural reason.

Ground-truth decay and the fixture management problem

A golden dataset built against live web content starts decaying the moment it's created. Without active management of the fixtures behind it, ground-truth decay quietly turns real regressions into false passes and real improvements into false failures, and the harness has no way to flag the difference on its own. The source page moves on, the golden answer doesn't, the eval keeps running, and the scores stay stable even though they no longer reflect the current state of the web. That stability is the danger: a flat scoreline looks like health when it's actually just staleness nobody caught.

Two fixture strategies address this, each with a real cost attached. The first is the frozen snapshot: capture the crawled page at the moment the golden is created, and replay that exact snapshot on every eval run afterward. This removes nondeterminism entirely, keeps the harness fast and cheap to run, and gives reproducible scores run over run. Its weakness grows quietly: the gap between the frozen snapshot and the live web widens every day the snapshot goes unrefreshed, until the harness is testing a version of the internet that no longer exists. The second strategy is live re-crawl with diff-gating: re-fetch the source at eval time, diff it against the original snapshot, and flag any golden whose source has materially changed so a human or an automated process can re-annotate it before scoring proceeds. This approach costs more in time and compute, but it keeps the harness honest about what it's actually measuring.

A baseline-store pattern, where each run's scores get saved and compared against prior runs with some configurable tolerance, gives a useful regression-detection layer on top of either strategy. It cannot, by itself, tell a genuine regression apart from a golden that has simply decayed. This is why the diff-gating step matters: it distinguishes "the agent got worse" from "the world changed and the golden didn't.

Good fixture hygiene makes both strategies workable in practice. Tag every golden with its source URL, the timestamp of the crawl that produced it, and a content hash of the source page. Trigger re-annotation automatically the moment that hash changes, rather than on a fixed calendar schedule, since page-change timing rarely lines up with a sprint cadence. Teams running recurring, unsupervised pipelines, lead triage, research summarization, content monitoring, carry the most exposure here, because errors from an undetected decay compound silently across every run of the pipeline before a human ever lays eyes on the output.

Freshness-aware scoring as a first-class eval criterion

An eval harness for web-grounded agents needs a freshness score sitting alongside correctness and groundedness as its own scored dimension, not as a side note buried in infrastructure. The question a freshness score asks is distinct from "is the answer supported by the source": it asks whether the source itself was current at the moment the agent retrieved it. A faithfully grounded answer can still be wrong if the page it leaned on was already stale by the time the agent fetched it. Grounding an answer against an outdated source is not the same thing as grounding it against the web as it actually stands.

A freshness score draws on three inputs. The first is the time delta between when the content was crawled and when the eval is being run. The second is any detected change in the content since the crawl, found by diffing the current page against the captured version. The third is the source's volatility class: a stock price page and a company's "About" page carry very different expected rates of change, and treating them the same wastes the signal either one could provide on its own.

Classifying sources by volatility lets the harness apply freshness thresholds that scale with how fast each source changes. High-volatility sources, pricing pages, breaking news, live data feeds, warrant much shorter acceptable staleness windows than low-volatility sources like documentation or legal terms, where content might reasonably hold steady for months.

The production pattern taking hold in 2026 pairs retrieval-augmented generation with a faithfulness judge that gates the final answer: if the judge's score falls below threshold, the agent re-retrieves or refuses. Freshness scoring adds a second, parallel gate to that pattern. Rather than catching only the case where the answer fails to match its source, it catches the case where the source itself, not the answer built on top of it, is the actual problem, and triggers re-retrieval specifically for that failure mode.

LLM-as-judge for web-grounded outputs

LLM-as-judge has become a standard tool for grading non-deterministic agent output, and it solves part of the evaluation problem that web-grounded agents create. It also introduces failure modes of its own that a well-built harness has to account for.

Deterministic functional assertions beat LLM-as-judge in several clear cases: task completion checks, schema validation, regex matching against structured fields. These checks run fast, produce the same result every time, and hold up to scrutiny in a code review the way a judge's score can't. Harness guidance from 2026 argues functional assertions should serve as the primary scoring mechanism precisely because they sidestep judge bias, cost less to run at scale, and produce a number a reviewer can defend without having to trust a second model's opinion.

LLM-as-judge earns its place in the harness where no deterministic check can reach: judging whether a free-form summary faithfully represents a crawled page, assessing whether a synthesized report reads coherently, deciding whether a retrieved answer actually responds to an open-ended instruction. These are judgment calls with no fixed structure to validate against, which makes them the territory where a language-model judge adds real value.

Groundedness scoring needs a specific constraint to make LLM-as-judge trustworthy: decompose the response into individual claims and score each claim's support separately. A holistic rubric misses unsupported claims that a well-written response is good at burying among several well-supported ones. Claim-level scoring catches what the holistic version lets through.

A few configuration choices limit how much the judge itself can contaminate the results. Using separate API keys for the target model and the judge model keeps the two from being the same system grading itself. Reporting safety metrics on their own, rather than folding them into a composite score, keeps a high overall average from masking a safety failure that would otherwise go undetected. The working principle for the whole harness: run deterministic assertions first, and bring in LLM-as-judge only for the portion of the output that deterministic checks structurally cannot cover.

Trajectory evaluation for multi-step web retrieval chains

Trajectory evaluation is the only mechanism that catches the specific failure where a final answer looks correct while the retrieval and reasoning path behind it is actually broken.

A web-grounded agent's trajectory includes which URLs it chose to crawl, which extraction schema it applied to the content it pulled, which retrieved chunks it selected to build its answer from, and how it assembled claims out of those chunks. Each of those steps is a separate point of failure, and each one can go wrong independently of whether the final answer happens to land correctly anyway. An agent can crawl the wrong page, extract the right fact from it by coincidence, and still produce a correct-looking answer that an outcome-only eval would wave through.

Three metrics give a harness visibility into this path: PlanAdherence checks whether the agent followed the retrieval plan it should have followed, StepEfficiency checks whether it took unnecessary tool calls along the way, and ToolCorrectness checks whether it invoked the right tool with the right inputs at each step. Multi-step agent workflows hallucinate across a meaningful fraction of their tool-call chains. A harness scoring only final outputs will therefore pass a large share of runs that are actually broken underneath a correct-looking surface.

Research into harness design backs up why this layer deserves investment. The Evo-Bench work from Huang et al. (2026) shows that improvements made to the harness itself produce large absolute performance gains even when the underlying model's strength is held constant. Trajectory design is a lever that works independent of the model sitting underneath it. Trajectory evaluation is therefore testing something a team can actually improve without needing to swap in a stronger model.

Benchmark infrastructure is starting to move in this direction on its own. WebArena-Verified rechecks task descriptions, reference answers, and evaluators by hand, and replaces nondeterministic LLM-as-a-judge scoring and substring matching with type-aware normalization and structural comparison wherever that's possible. That's a direction internal harnesses should take seriously: wherever a deterministic structural check can replace a judged one, it should.

Test environment design: when to use live crawls, frozen snapshots, and self-healing fixtures

No single test environment mode covers every stage of the eval lifecycle. A harness that's both fast enough to run in CI and honest enough to catch real failures uses live crawls, frozen snapshots, and self-healing fixtures at different points, not one of the three everywhere.

During development, frozen snapshots are the right default. The same page state replays on every run, scores stay reproducible run to run, and a developer can run the full suite against a pull request without paying crawl cost or dealing with the nondeterminism a live fetch would introduce. Speed matters here more than freshness, since the goal at this stage is fast iteration, not a final verdict on production readiness.

At merge time in CI, a diff-gated refresh earns its cost. The system checks whether any source page has changed since its snapshot was captured. Where a source has changed, that golden gets flagged for re-annotation before the suite runs against the new state. This step is what keeps a silently decayed golden from slipping onto the main branch disguised as a passing test.

In scheduled production evals, live re-crawl paired with freshness scoring gives the signal that actually matters at that stage: the agent evaluated against the web as it exists right now, with freshness metrics flagging any answer that leans on content that has since moved on. Each of the three modes serves a different purpose in the lifecycle, and a harness that picks only one of them is choosing either speed, safety, or truth, when the real job is holding all three in the right place at the right time.