Coding Agent Trajectory Evaluation Assess

coding-agentsevaluationagentsagent-evaluationtrajectory-analysisobservabilitydeveloper-experiencebenchmarkingllm-opsreliability

Overview

Coding Agent Trajectory Evaluation is the practice of scoring and diagnosing an agent run by its whole trajectory — the searches, reads, edits, tool calls, validation, reversions, and messages to the user — rather than by the single bit of whether the final patch passed the target tests. AgentLens frames the motivation plainly: most code-agent benchmarks reduce a run to one bit, but the people who actually use these agents experience the entire trajectory, including how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way (AgentLens). The same argument appears in the broader agent-evaluation literature, where log analysis is presented as necessary to overcome validity threats: scores inflated or deflated by shortcuts and benchmark artifacts, benchmark performance that fails to predict real-world utility, and capability scores that conceal dangerous actions (log analysis).

The adjacent operating argument matters as much as the measurement one. The coding-agent reliability monograph observes that AI coding agents are commonly evaluated as models but deployed as systems, with reliability depending on the harness, execution state, retrieval, memory and state management, permissions, review interfaces, and resource allocation (Engineering Reliable Coding Agents). Trajectory evaluation is the natural instrument for that system view, because the harness and state behaviour show up in the trace long before they show up in a resolve rate.

It sits in assess because the direction is well argued and directly relevant to anyone operating coding agents, but production practice is unproven. The published frameworks are recent preprints with essentially no citation history, the vocabulary is still being disambiguated, and evidence of sustained production use is limited to authors evaluating their own agents. Assess means: read the papers, instrument one pipeline, and learn what your own traces tell you — not standardise on a scoring scheme yet.

Adoption Signals

  • AgentLens is released as an open-source, production-assessed benchmark for interactive code agents, pairing formal verification (where an objective check exists) with LLM-written trajectory reviews and side-by-side comparisons (AgentLens).
  • Its authors report using it beyond ranking: to diagnose model behavior, compare successive versions of their own agent, and catch product regressions in a nightly evaluation pipeline (AgentLens).
  • TraceProbe has been applied to 2,500 trajectories from five production settings on SWE-Bench Verified, finding that file choice is too coarse to separate success from failure while function selection and completion behavior localise it (What Resolve Rate Hides).
  • TrajEval decomposes trajectories into reference-patch-aligned search, read, and edit stages across 16,758 trajectories, three architectures and seven models, and reports that 60–69% of SWE-Agent and OpenHands failures reach and edit the correct functions yet still produce incorrect patches (TRAJEVAL).
  • Tooling is appearing alongside the research: Datadog Labs' trajectory captures coding-agent sessions across many agent CLIs and turns them into timelines, traces, operational metrics, and YAML-defined "Markers" for agent behaviour (datadog-labs/trajectory).
  • Reviewability of delegated agent work is itself becoming a measured quantity, via software delegation contracts covering the task, authority, returned work package, and acceptance context (Software Delegation Contracts).

Risks

  • Immature and unsettled measurement vocabulary. Existing process-level evaluations often treat action prediction, task uncertainty, and step attribution as if they were the same thing, when they are three different levels (What Process Evaluation Actually Measures).
  • Trace heterogeneity. Raw traces are heterogeneous and hard to compare across runs, which is exactly why TraceProbe has to normalise each run into a canonical nine-type action taxonomy with deterministic effect labels before diagnosing anything (What Resolve Rate Hides).
  • Weak diagnostics can mislead. TraceProbe finds that some anti-patterns act mainly as corpus-level difficulty clues rather than per-run verdicts, so naive anti-pattern counting risks confident but wrong conclusions (What Resolve Rate Hides).
  • LLM-judged reviews carry cost and subjectivity. AgentLens deliberately combines formal verification with LLM-written reviews because objective checks do not always exist, which means part of every score rests on model judgement (AgentLens).
  • Evaluation is not a security control. Trajectory data touches sensitive ground — agents can exhibit unsafe behaviours during normal operation that manifest as vulnerabilities (insecure behaviors) — and reliable operation still requires the surrounding permissions, sandboxing and audit controls (Engineering Reliable Coding Agents).
  • Static evaluation drifts. Guidance on agent evaluation stresses a closed loop from offline eval to CI gate to production trace eval and back; treated as a static endpoint, the eval goes stale while the agent moves (LLM agent evaluation guide).

Pros & Cons

Advantages

  • Trajectory reviews yield a readable explanation of why a run scored the way it did, which makes evaluation results usable for diagnosis rather than only for ranking models.
  • Process-level diagnostics surface failure modes that a pass/fail label hides, such as agents that reach and edit the correct functions yet still produce an incorrect patch.
  • Trajectory evidence can be wired into routine engineering practice, for example a nightly evaluation pipeline that catches product regressions between successive versions of an in-house agent.

Disadvantages

  • Raw traces are heterogeneous and hard to compare across runs, so teams must invest in normalization (canonical action taxonomies, effect labels) before any diagnostic is meaningful.
  • Terminology is unsettled: process evaluation conflates action prediction, task uncertainty, and step attribution, which are three different levels of measurement.
  • Parts of the approach depend on LLM-written trajectory reviews and side-by-side comparisons, adding judge cost and subjectivity on top of formal verification where an objective check exists.

Recommendation

Start by capturing trajectories, not by choosing a scoring rubric. Turn on session capture for the agents your teams already use and get to the point where you can answer basic operational questions — where time, tokens and cost went, and whether the agent made progress, looped, retried, or stalled — before you attempt any judgement of quality; off-the-shelf tooling such as datadog-labs/trajectory is a reasonable starting harness. Normalising traces into a canonical action taxonomy with effect labels, as TraceProbe does, is the prerequisite for comparing runs at all.

Then run one narrow pilot on an internal agent or workflow. The most credible published pattern is AgentLens's: use formal verification wherever an objective check exists, add trajectory review only where it does not, and point the pipeline at regression detection between successive versions of your own agent rather than at cross-vendor leaderboards (AgentLens). Pick a small number of hypotheses your traces can actually settle — for example whether your failures are localisation failures or edit-quality failures, the distinction TrajEval draws (TRAJEVAL) — and be explicit about which level you are measuring, since action, task and step are not interchangeable (process evaluation levels).

Do not make trajectory scores a merge gate or a vendor-selection criterion this quarter. Treat them as diagnostics that inform the system around the model — harness, retrieval, memory and state management, permissions, review interfaces, resource allocation — which is where the reliability monograph locates most of the leverage (Engineering Reliable Coding Agents). If reviewability of delegated work is your pressing concern, read the delegation-contract framing alongside it (Software Delegation Contracts). Revisit the ring next release once you have a quarter of your own trace data.

Sources