The Agent Waste Index

The one-page version executives, finance, operators

We took every public dataset of AI coding-agent runs we could find — 341,054 runs from 11 datasets, spanning 2024 Llama SWE-agent runs, GPT-4o, Claude 3.5 and 3.7 Sonnet, Qwen3-Coder-480B and gpt-5-nano, on four scaffold families — and asked one question: where does the money go, and does it buy anything?

Three things follow, in plain language:

1. The classic failure, an agent stuck repeating itself, is mostly a weak-model artifact. Claude 3.7 Sonnet loops in 0.4–0.5% of runs and blindly retries in under 1%; Qwen3-Coder-480B stays under 1.5% on both. The 2024 Llama SWE-agent runs: 9–18% loops, 22–31% blind retries. The exception is gpt-5-nano on terminus-2 (5% and 5%), where a median run is five steps and one repeat is a large share of it. Mechanical waste totals 4% of estimated spend across the whole set.
2. The scaffold the agent runs in leaks more money than the model does, and the cheap one solves less. One Open-SWE trace version, same tasks, three scaffolds: two dragged oversized tool output through 50% and 55% of runs; the bash-only one, 0.1%. With that version's outcome labels now available, the bash-only scaffold resolves 30% of tasks against SWE-agent's 53%: still cheaper per solved task on the estimated basis ($8.61 against $11.88), but far fewer solved. Choosing on cost alone is choosing a lower success rate.
3. Nobody is measuring the value of the next step. Success rates fall steadily as runs get longer while each step costs more than the last. By the last fifth of a run, a dollar buys between an eighth and a 285th of what the first dollar bought. Teams set a hard cap or no cap; almost none measures the trade-off.

What it means for the budget finance

You don't need to understand agents to understand this: a growing line item where two-fifths of the money buys the fewest results is a procurement problem, not a technology problem.

• The spend is concentrated and controllable. The longest fifth of runs takes 21–72% of a group's estimated spend (median 50%, 40% across the whole set). That concentration is a property of how agents work (the context grows, so every step costs more than the last), not of any one vendor.
• Run length is a lever nobody has measured. Unlike model choice or vendor pricing, how long a run is allowed to go is something your own team controls today. A cap still pays for every step up to the cap, so the trade-off has to be tested, not assumed.
• Iteration caps are already burning money silently. One configuration in the data (GPT-4o on OpenHands, 30-iteration limit) cut off 53% of its runs; those runs took 55% of the group's estimated spend and resolved 1% of the time. A cap set wrong costs as much as no cap.
• Waste from "dumb" failures is small for modern agents — 4% of estimated spend across the set, 0–10% in the modern groups, and 23–38% only on the 2024 Llama agents — so a tool that only finds loops will not pay for itself. The money is in run policy, framework choice, and outcome tracking.

Three questions for your AI or engineering lead:

1. What share of our agent spend last month went to runs that produced nothing? The data to answer it exists in every trace; most teams have never computed it.
2. What limits do our runs have, and has anyone measured what those limits cost in solved tasks?
3. Which agent framework are we on, and did anyone compare its cost profile to an alternative?

What it means for whoever owns the agents heads of AI, operators

Claude 3.7 Sonnet, 14,374 runs, tool-call format. Runs split into five equal groups by length. Bars: share of estimated spend with cached input pricing. Line: share of runs that resolved the task. Resolve falls from the shortest to the longest fifth in 17 of the 18 groups that carry resolve labels; the exception is GPT-4o under a 30-step cap, whose shortest runs resolve nothing at all.

• Treat run length as a first-class measurement. Compute your own resolve-by-length curve from traces you already keep before touching any limit, and test a limit the way you would test any other change: on the same tasks, against the baseline.
• Choose the scaffold on cost, not just capability. Oversized tool output (a single result over 20,000 characters, then re-sent every step) hit 20–22% of Claude 3.7 runs and 10–16% of Claude 3.5 runs on SWE-agent, 7–72% of runs on OpenHands, and at most 0.2% on bash-only mini-SWE-agent. Bounding tool output is the most common scaffold-level fix the data points at, and the easiest to test.
• Check your caps against your resolve curve. Either the cap is too low for the task or the task is wrong for the agent; both are visible in the curve.
• Stop instrumenting for loops. They are near-zero for frontier models and a symptom of a weak model or a very short run otherwise. Instrument for outcome, length, and tool-output size.
• Early signals are weak once you control for length; length itself is the signal. A big tool result early in a short run predicts failure (Claude 3.7: 50% vs 65% resolve in the shortest quintile). Beyond that the flagged-vs-clean gap shrinks to a few points, and across the full set it reverses in 5 of the 23 groups with resolve labels. Build the stop/escalate decision on expected value per step, not on detector hits, and see the retraction below before shipping a mid-run kill switch.

Findings, in full engineers, researchers

Headline table

Loops: identical action three or more times with no state-changing action between. Blind retries: identical action immediately after an error observation. Big output: any single observation over 20,000 characters. Ended without result: context exhausted, iteration or cost limit, or cut off mid-call. Mechanical waste: cost of loop and retry steps plus the cost of dragging oversized output through remaining steps. Resolve: task passed hidden tests where the dataset records it. Cost basis: characters/4, full context re-sent each step, Sonnet-class rates; quote the percentages. Groups under 1,000 runs omitted; everything is in index.json. Ingest is capped at the first 20,000 rows of each dataset/config/split in the publisher's own streaming order, so the larger sources are a head sample, not the whole dataset. Edit thrash (the same file edited six or more times) is reported in the repo tables but is not counted as mechanical waste and not counted as a finding: it fires on over 30% of runs in 20 of the 29 groups and peaks at 78%, which makes it a description of how a scaffold edits files rather than a detection of waste. The agentic-coding-trajectories rows are that dataset's own provenance labels and resample populations already in the set — read them as replication, not as extra independent evidence, and do not pool them with their sources.

Same trace version, same tasks, three scaffolds

nvidia/Open-SWE-Traces. The dataset card does not attribute a generating model per config and the traces carry no model field, so rows are labelled by trace version and compare scaffolds within a version, never models. The card describes the set as completed submissions curated for SFT, and says v1.0 was filtered after release to remove "git hacking" runs. We can confirm the consequence but not the cause: failed-run cost measures 0% across all six groups.

Bash-only runs are a third shorter with zero oversized output; the scaffold eliminates a waste class by construction. Its long tail is wider, so its spend is more concentrated in the longest fifth, not less. Since edition 1.1 the v1.1 sweagent and minisweagent groups and the v1.2 group carry outcome labels (v1.1 openhands still does not): on v1.1 the bash-only scaffold resolves 30% against SWE-agent's 53%, which on the estimated basis is $8.61 against $11.88 per resolved task. Cheaper per solved task, far fewer solved. The v1.2 bash-only group resolves 57% at $6.82, so the gap is a property of that version's tasks and model as much as of the scaffold. Quote the pair, never the scaffold alone.

Cost and resolve by run-length quintile

Within each group, runs are split into five equal-count buckets by step count. Two cost bases: full context re-sent every step, and cached input (re-read tokens at 10%). Resolved per $ is relative within a group. Groups with resolve labels and at least 500 runs per bucket.

What we retracted along the way

Five claims from earlier versions of this work did not survive the full run, or a re-run. They're recorded so nobody re-derives them from an old version of the tools, or from an old version of this page.

• "Waste predicts failure everywhere" — retracted. The 4,000-run pilot found that runs tripping any detector resolved at roughly half the rate of clean runs, in all five groups it covered, and pitched that as a mid-run kill signal. At 341,054 runs it does not generalise: of the 23 groups with resolve labels, 18 resolve worse when flagged and 5 resolve better. The effect is large only on weak models (swe-agent-llama-8b 0.16×, llama-405b 0.19×, llama-70b 0.21×, gpt-4o on OpenHands 0.25×) and it reverses on the SWE-smith Claude 3.5 Sonnet groups (1.10×, 1.21×, 1.22×) and mini-coder-trajs-400k (1.18×). What survives is narrower: a tripped detector is a strong failure signal on weak models and close to no signal at the frontier. Do not ship a mid-run kill switch on this basis without measuring your own traffic first.
• Parallel tool calls under-read — corrected in edition 1.1. Edition 1's parser attached only the first tool result of a turn that issued several calls, so the later results, and any oversized output or error they carried, were invisible to the detectors. Fixed and re-swept in September 2026; the aggregate moved by a twentieth of a point because these scaffolds mostly issue one call per turn. Recorded because traces from a current coding agent are not like that, and an edition 1 local audit of such traces under-counted. Three Open-SWE groups also gained outcome labels between editions, and two mini-swe-agent groups lost their cut-off runs; neither is the parser fix, and the method notes say what we could and could not establish.
• Pilot dollar figures — withdrawn. The pilot quoted "$8–9 per failed Claude 3.7 run" and "22% of Claude 3.7 spend on runs that produced nothing". Both came from a 2,000-run slice with no exit status, where "produced nothing" was a no-submit heuristic. With the full splits parsed and end-of-run rules applied, SWE-smith Claude 3.7 runs end submitted or with a closing message, and the failed-run share for those groups is 0%. Those per-run dollar figures should not be quoted at all.
• Model labels for nvidia/Open-SWE-Traces — withdrawn. An earlier edition of this page named specific generating models for the three Open-SWE configs. The dataset card attributes none and the traces carry no model field, so those labels were never verified; they have been replaced with the trace version. Three configs, six groups and 120,000 runs ride on that label, and it is better read "see card" than read wrong.
• "Frontier agents loop in 67% of runs" — retracted. An artifact of treating every repeated command as a loop. Edit, test, edit, test is not a loop. With the state-change reset, frontier loop rates are 0–2%, and the edit-thrash detector that produced much of that number now sits outside mechanical waste entirely, for the reason given under the headline table.

Method, data, caveats

Sources

Caveats, stated plainly

• Groups are head samples. Ingest is capped at the first 20,000 rows per dataset/config/split, taken in the publisher's own streaming order. For the largest sources (mini-coder-trajs-400k, SWE-ZERO-12M-trajectories) that is the head of the dataset, not the dataset, and it inherits whatever ordering the publisher used.
• Model labels are labels, not attributions. Open-SWE rows are labelled by trace version. nebius-swe-rebench-openhands, swe-smith-claude-3-7-sonnet and kwai-klear-swe-smith-mini are the source labels thoughtworks/agentic-coding-trajectories carries for its own rows; the first matches nebius/SWE-rebench-openhands-trajectories to within sampling noise on every statistic, so treat it as a resample of a population already here rather than as independent evidence.
• Edit thrash is reported, not costed. The same file edited six or more times fires on over 30% of runs in 20 of the 29 groups and peaks at 78%. The detector cannot tell a productive edit-test-edit cycle from thrashing, so it is excluded from mechanical waste and from "any finding".
• Dollars are estimates. No dataset carries per-run cost. Tokens are characters/4; context is modeled as re-sent each step at Sonnet-class rates, with and without cached input pricing. Percentages and orderings are what we stand behind.
• These are training-data and benchmark runs, not production. Most are curated, and what we know about how comes from the dataset cards rather than from the traces: SWE-smith kept useful runs, Open-SWE kept complete submissions and filtered out "git hacking," Nebius filtered. Take those as the publishers' descriptions. Resolve rates are not comparable across datasets; within-dataset comparisons are.
• Frontier coverage is thin. The strongest models with outcome labels are Claude 3.7 Sonnet and GPT-4o. The three Open-SWE configs — 120,000 runs — carry no model attribution at all, and six of the 29 groups carry no resolve labels. The one newer-model set here — gpt-5-nano on terminus-2, 20,000 runs — carries no resolve labels, so it contributes to the cost and detector numbers and to none of the outcome ones. There are no public trajectory sets at all for Claude 4.x or Gemini 2.5, and the SWE-bench leaderboard bucket and Princeton's HAL traces are access-restricted. The patterns hold from a 1.7B model to Claude 3.7, which is why we expect them to hold, but that is an expectation.
• End-of-run classification is per-scaffold and heuristic where the dataset carries no exit status. The full terminal × exit-status table ships in index.json.
• Selection. Long runs are partly long because tasks are hard. The finding is about marginal value per step, not that capping is free.

Reproduce

git clone https://github.com/metermaidai/audit && cd audit
python -m pip install datasets pyarrow duckdb
python pipeline.py sweep --limit 20000     # streams every registered dataset into data/runs/*.parquet
python pipeline.py report                  # DuckDB over Parquet -> report/index.md, index.json

Run it on your own agents

Everything above came from public runs. The same detectors, plus the spend-side audit (over-tier models, missing prompt caching, batch-eligible jobs, unowned keys), run locally against your own traces and your Anthropic or OpenAI admin API. Nothing leaves your machine.

python metermaid_audit.py --share          # spend audit; share.json is the only file meant to leave
python trajectory_audit.py ./your-traces --share   # loops, retries, big output, restarts, ended without result

Send share.json, the allowlisted export, and get your findings, your resolve-by-length curve and your position against the Index. The next edition adds contributed production data, which is the only way current frontier models get into it.

The Agent Waste Index, edition 1.1. metermaid, September 2026. Data, code, and every retraction are in the repo.