metermaid finds avoidable cost across your agent workflows — repeated calls, oversized tool output, restarts, missed caching, the wrong model on the wrong task — helps your engineers test the fixes, and measures what changed. Start with a free audit that never leaves your machine.
Oversized tool output, re-sent every step. 55% of SWE-agent runs on Open-SWE v1.1 carry a single tool result over 20,000 characters. The same tasks on a bash-only scaffold: 0.1%.
metermaid produces this split for every task type from your own billing data and traces, joined to outcomes. The shares here are measured on one group of the Index — Qwen3-Coder-480B on OpenHands, 20,000 public runs, cached pricing basis. Yours will differ, and that difference is where the money is.
The first four fifths of runs, minus the waste below. Where nearly all the resolved tasks are, and the only bucket you want to be paying for.
Loops, blind retries, oversized tool output re-sent every step. 4% of estimated spend across the whole Index — cheap to find, cheap to fix, and the first thing we show you.
Hit a cap, exhausted context, cut off mid-call. Produced nothing. 0% in the curated Claude training sets, 55% for GPT-4o under a 30-step cap, 98% for a 1.7B model.
The longest fifth of runs: 21–72% of a group's estimated spend, returning 8–285× fewer resolved tasks per dollar than the shortest fifth.
The three waste shares are measured separately and can overlap slightly — a long run can also be a failed run — so the bar draws them as disjoint and the first bucket is the remainder. On your own data the split is exclusive, computed per run.
What 341,054 public agent runs across 11 datasets, from 2024 Llama agents to Claude 3.7 Sonnet, say about where the money goes. Public, reproducible, every retraction and correction included — five so far.
of estimated spend goes to the longest fifth of runs across the whole set (37% under cached pricing) — 50% for the median group, and up to 72% in a single one. Resolve falls from the shortest fifth to the longest in 17 of the 18 groups that carry resolve labels — to well under half the rate at the median (0.42×), and anywhere from 0.15× to 0.91× by group.
of Claude 3.7 Sonnet and Qwen3-Coder-480B runs loop or blindly retry. In 2024-era open models it was 9–31%. The classic failure is a weak-model problem now.
of runs dragging oversized tool output, same trace version, same tasks: bash-only scaffold vs SWE-agent. The bash-only one also resolves 30% against 53%: cheaper per solved task, far fewer solved.
Claude 3.7 Sonnet, 14,374 runs, uncached basis. Resolve falls from the shortest to the longest fifth in 17 of the 18 groups that carry resolve labels — Claude 3.5, Qwen3-Coder, Llama, Open-SWE v1.0, v1.1 and v1.2. The exception is GPT-4o under a 30-step cap, whose shortest runs resolve nothing at all.
Success rates fall steadily as runs get longer, while every step costs more than the last because the context grows. By the last fifth of a run, a dollar buys between an eighth and a 285th of what the first dollar bought, depending on the model.
Most teams set a hard step cap or none, and almost none measures the trade-off. A cap still pays for every step up to the cap, so it is a test to run, not a saving to assume. The free audit computes this curve from your own runs; the pilot tests the fixes it points at.
One workflow, your own data, a measured result.
Read-only admin keys for Anthropic and OpenAI, run from your own machine. Trace files from SWE-agent, mini-swe-agent, OpenHands, or plain message lists. Outcomes from whatever your workflow already records: a test result, a merge, one boolean per run. Nothing is uploaded unless you send share.json.
Cost per successful task with every attempt counted, restarts included. Your resolve-by-length curve. Failed-run cost. Where oversized tool output is being re-sent, where retries repeat the same failing call, where caching is missed. Spend nobody owns.
Up to three changes chosen for your workflow: bounded tool output, retry recovery, prompt caching, restart reuse, and batch or model experiments where your data says they are worth it. Your engineers make the change; we prepare it when that helps.
Baseline against each fix and the combined configuration, on the same tasks, with quality, coverage and uncertainty shown. Individual fixes are never added up: the combined number is the claim, and a negative or inconclusive result is reported as one.
The audit and the Index pipeline are open source and stay that way. You pay for the pilot that tests fixes on your workflow and measures the result, and for the checks that keep the savings when models, prompts and scaffolds change.
share.jsonWe recommend the pilot only when the free audit finds an opportunity worth several times its price, and continuous review only when the verified saving is several times the fee. If neither is true we say so, and you keep the audit.
Run it yourself and nothing leaves your machine. Send us share.json, the allowlisted export with pseudonymised ids and no commands or run ids, and we come back with the findings, your run-length curve, and where you sit against 341,054 public runs. Or leave your email and we set the audit up with you.
One reply from a person. No newsletter. Nothing leaves your machine unless you send share.json.
# clone and install
git clone https://github.com/metermaidai/audit
cd audit && pip install requests
# spend audit: read-only admin API, runs locally
export ANTHROPIC_ADMIN_KEY=sk-ant-admin-...
python metermaid_audit.py --share
# trace audit: loops, retries, big output, restarts
python trajectory_audit.py ./your-traces --share
# share.json is the only file meant to leave your machineNot with the CLI. It reads your provider's admin API or your trace files locally and writes a report to disk. The only file meant to leave your machine is share.json, built by allowlist: totals, shares, per-detector counts and pseudonymised ids, never run ids, commands, prompts, tool output, or owner emails. It lists inside itself what it never contains, and the tool prints what it holds before writing it.
Because finding is the easy part. The pilot is where the value is: a baseline, up to three fixes tested on your workflow, and the combined result measured with quality included. Continuous review is for keeping those savings when models, prompts and scaffolds change. If one audit fixes your problem for good, you weren't a customer, and we're glad it helped.
Those are the strongest models in the set that carry outcome labels, alongside Qwen3-Coder-480B. Newer entries arrive without them: gpt-5-nano has no resolve labels, and the 120,000 Open-SWE runs carry no model attribution on the dataset card at all, so we label them by trace version rather than guess. Leaderboard and academic trace archives for newer models are access-restricted. The patterns hold from a 1.7B model to Claude 3.7 Sonnet, which is why we expect them to hold, but that is an expectation. Contributed production runs are how the next edition gets there.
Tracing tools show engineers calls, tokens, and latency. FinOps tools show finance the invoice. Neither ties cost to a completed task with every attempt counted, and neither tests a fix against a baseline on the same tasks. metermaid reads trace files those tools can export and does the comparison.
No, and every table says so. Public datasets carry no per-run cost, so dollars are estimated from characters at Sonnet-class rates. The percentages and orderings are what we stand behind. Your own audit prices from your traces' recorded cost or your provider's usage and a dated rate card, and says which.