An optimization pilot ends with one document: where the workflow's money went, which fixes were worth testing, and what each fix and the combination did to cost and quality on the same tasks. This is that document, built from 60,000 public runs instead of a customer's, so you can see the format before you send anything.
Before any fix is proposed the report states what was measured, how much of the data it covered, and what it could not see. Everything later in the report is relative to this table.
| Workflow | Open-SWE v1.1 on SWE-agent (the scaffold the tasks were first run through) |
|---|---|
| Window | First 20,000 runs of the split, as served by the dataset in September 2026 |
| Intake | 20,000 records parsed, 0 files not parsed, 0 duplicate attempt ids dropped |
| Tasks and attempts | One attempt per task in this data, so task cost equals run cost. A customer's pilot counts every attempt at a task, restarts included, and reports how many tasks were restarted. |
| Outcomes | 17,764 of 20,000 runs carry a label (field: resolved); 2,236 are unknown and stay unknown. Success rate 53% over labelled runs. |
| Median run length | 76 steps |
| Cost basis | Estimated, characters ÷ 4 at Sonnet-class rates, with and without cached-input pricing. Rate card version 2026-09-15. |
| Cost per successful task | $11.88 on the estimated basis, every labelled run in the numerator |
The Intake and Outcomes rows are the ones customers skip and should not. If 40% of your runs carry no outcome, the resolve figures below describe the other 60%, and the report says so rather than treating missing as failed.
Runs sorted by step count and cut into five equal buckets. Each row is that bucket's share of estimated spend, its success rate, and how many successful tasks a dollar bought.
| Bucket | Steps | Share of spend | Share, cached basis | Success | Successes per $ |
|---|---|---|---|---|---|
| 1 (shortest) | 19–63 | 11% | 13% | 67% | 0.172 |
| 2 | 63–72 | 15% | 16% | 58% | 0.109 |
| 3 | 72–81 | 18% | 19% | 53% | 0.083 |
| 4 | 81–95 | 22% | 22% | 48% | 0.059 |
| 5 (longest) | 95–245 | 33% | 30% | 40% | 0.032 |
The longest fifth of runs takes a third of the spend and returns a fifth as many successes per dollar as the shortest fifth. That is a concentration finding, not a saving: a cap still pays for every step up to the cap, and the tasks in the long bucket may be the hard ones. It tells you where to look, and it is why ticket 2 exists.
| Detector | Runs affected | What it means here |
|---|---|---|
| Oversized tool output | 55% | A single tool result over 20,000 characters, re-sent in the context of every later step |
| Blind retry | 2.3% | The identical command re-issued immediately after it failed |
| Loop | 0.2% | The same action three or more times with nothing changing in between |
| Ended without a result | 0.0% | No submission, context limit reached, or cut off |
Costed together, the three mechanical detectors account for 2% of this group's estimated spend. That number is small on purpose: the detectors price only the waste they can see, and the report never adds the categories together as if they were independent.
Every finding that is worth a conversation becomes a ticket: the evidence, a proposed fix, who owns it, how it will be tested, and its status. Ranked by expected dollars, then by evidence, then by effort. Three is the ceiling for a pilot; a low-ranked ticket is still written down so the decision not to test it is on record.
55% of runs carry a single tool result over 20,000 characters, and the scaffold re-sends it on every later step.
The longest fifth of runs takes 33% of estimated spend and succeeds at 40%, against 67% for the shortest fifth.
2.3% of runs re-issue the identical command right after it failed.
Each fix is run on the tasks the baseline ran, alone and then all together. Cost counts every attempt. A fix that lowers the success rate by more than the agreed tolerance is rejected however much it saves. The combined configuration is the number the pilot claims; the individual fixes are shown so the overlap between them is visible, and they are never added up.
In this sample the only change that public data lets us measure is a scaffold swap, because Open-SWE v1.1 ran the same tasks through SWE-agent and through a bash-only scaffold. Treat it as a candidate fix named bash-only, tested against the SWE-agent baseline. This is the exact output format of compare.py; the numbers are group aggregates from the Index rather than a task-paired cohort, which the pilot version would be.
# metermaid comparison Baseline against 1 arm(s) on the tasks present in every arm. Cost counts every attempt at a task. Quality tolerance 2.0 points; minimum cohort 20 tasks. Latency is not in trajectory records and is not reported. ## Result **REJECTED** — bash-only cuts estimated cost by 60% and fails the quality gate. | arm | role | success | Δ pp | unknown | $/success | Δ cost | status | |-----------|-----------|--------:|-------:|--------:|----------:|-------:|----------| | baseline | baseline | 53% | — | 2,236 | $11.88 | — | — | | bash-only | candidate | 30% | -23.7 | 2,292 | $8.61 | -60% | rejected | ### bash-only: rejected - success rate fell 23.7 points, beyond the 2.0-point tolerance - cost per successful task fell from $11.88 to $8.61 on the estimated basis - runs are shorter (median 51 steps against 76) and carry no oversized output ## Coverage - bash-only: 20,000 runs, outcome labels on 17,708
This is what an honest negative looks like. The cheaper scaffold solves fewer tasks, so a team choosing on cost alone would be choosing a lower success rate. The report says so and the ticket closes as rejected. In a customer pilot the same table would carry ticket 1, ticket 2 and the combined arm, with a 90% interval on the cost change from a paired bootstrap.
share.json: totals, shares, the five run-length rows, per-detector counts and pseudonymised ids.