Benchmarks
Nine measurements, agreed up front so results could not be chosen after the fact to suit a conclusion. Every run is reported per model. Nothing is averaged across models, because a client locked to one model needs that model's number.
3-repo ticket
2 runs · clean path, nothing injected
| Benchmark | Sonnet 5 fixed boundary | Fable 5 pre-boundary |
|---|---|---|
| Task completionDid the task finish, end to end. | complete3 cross-linked draft pull requests | complete3 cross-linked draft pull requests |
| Stops per runHow many times the run halted and handed back to a person. The number the study exists to reduce. | 0 | 0 |
| Backtracks per runReturns into an already-finished repository. Across repositories only; review cycles inside one repository are a separate, solved mechanism. | 0none needed | 0none needed |
| Correct backtrack rateOf those returns, how many targeted the repository that caused the problem. | not measured: no backtracks occurred, so nothing to score | not measured: no backtracks occurred, so nothing to score |
| Loop / thrash rateRepeated work without progress, instead of recognising the circle and stopping. | noneno review cycles at all; every repository passed its checks and its review on the first round | noneone write-then-check iteration per repository; no review cycles |
| Time to recoveryFrom an injected fault back to productive progress. | not measured: no fault injected in this run | not measured: no fault injected in this run |
| Context efficiencyHow much is carried between steps versus used. Token volume across dispatches is the proxy. | 606,417 tokensacross 8 top-level dispatches; ≈1,968,227 including everything nested beneath them | not measured: not captured: this column predates the fixed measurement boundary, so per-dispatch token totals are not comparable and were not recorded |
| Validation pass rateDependency edges checked and satisfied. | 3/3all edges satisfied at the gate; the third was the deliberately unstated edge, closed by what Compose declared it used | 3/3all edges satisfied at the gate |
| End-to-end latencyWall-clock for the whole workflow. | 1h 49m 21smeasured | ≈45–60 mincoarse estimate, not stopwatch-timed |
Sonnet 5 · TB-201 · run detail
Pull requests
tb-moby#3 · tb-cli#5 · tb-compose#5 (private test repositories)
Notes
- First column collected under the fixed measurement boundary: the session acted only as operator (seeding, resets, re-dispatch, logging) and a single agent ran the entire pipeline on this model, with every nested step inheriting it.
- Seeded-drift probe run separately against this ticket: stopped deterministically, at the same point and naming the same repository as the other model.
Fable 5 · TB-101 · run detail
Pull requests
tb-moby#1 · tb-cli#3 · tb-compose#4 (private test repositories)
Notes
- Collected before the measurement boundary was fixed: the coordinating role was played by the session inline rather than by a single agent on this model. Treat small differences against the baseline column as method, not model.
- Seeded-drift probe run separately against this ticket: stopped deterministically at the designed point, naming the correct repository.
- Two repository runs were killed mid-flight by session usage limits and resumed losslessly on re-dispatch, the evidence behind finding F4.
2-repo ticket
2 runs · clean path, nothing injected
| Benchmark | Sonnet 5 fixed boundary | Fable 5 pre-boundary |
|---|---|---|
| Task completionDid the task finish, end to end. | complete2 cross-linked draft pull requests | complete2 cross-linked draft pull requests |
| Stops per runHow many times the run halted and handed back to a person. The number the study exists to reduce. | 0 | 0 |
| Backtracks per runReturns into an already-finished repository. Across repositories only; review cycles inside one repository are a separate, solved mechanism. | 0none needed across repositories; 2 review cycles within repositories, one each | 0none needed across repositories; 1 review cycle within a repository |
| Correct backtrack rateOf those returns, how many targeted the repository that caused the problem. | not measured: no cross-repository backtracks occurred | not measured: no cross-repository backtracks occurred |
| Loop / thrash rateRepeated work without progress, instead of recognising the circle and stopping. | none2 review cycles, both on genuine defects, both resolved inside the bounded cap on the second round | none1 review cycle on a genuine defect, resolved on the second round |
| Time to recoveryFrom an injected fault back to productive progress. | not measured: no fault injected in this run | not measured: no fault injected in this run |
| Context efficiencyHow much is carried between steps versus used. Token volume across dispatches is the proxy. | 363,641 tokensacross 6 top-level dispatches; nested totals not independently surfaced for this run | not measured: not captured: this column predates the fixed measurement boundary, so per-dispatch token totals are not comparable and were not recorded |
| Validation pass rateDependency edges checked and satisfied. | 1/1the single edge satisfied at the gate | 1/1the single edge satisfied at the gate |
| End-to-end latencyWall-clock for the whole workflow. | 1h 31m 41smeasured | ≈30–45 mincoarse estimate, excluding an unrelated service outage |
Sonnet 5 · TB-202 · run detail
Pull requests
tb-moby#4 · tb-cli#6 (private test repositories)
Notes
- The review pass caught two genuine defects here, one in each repository: a new timeout leaking into unrelated calls, and a code comment pointing at a file that is never committed. The other model's equivalent run caught only the second.
Fable 5 · TB-102 · run detail
Pull requests
tb-moby#2 · tb-cli#4 (private test repositories)
Notes
- Collected before the measurement boundary was fixed. See the caveat on the other column from this model.
- This is the run where review caught a flag wired into a constructor with no production callers: all unit tests passed and the feature did nothing.
How the columns were collected
- fixed boundary One agent ran the whole pipeline on this model, every nested step inheriting it. The operator only seeded, reset and logged. Columns collected this way are directly comparable.
- pre-boundary Collected before the measurement boundary was fixed, with the coordinating role played inline by the session rather than by an agent on this model. Read differences as method, not model.
Model strategy
Which model the pipeline runs on changes the results, so it is measured and reported per model rather than blended into an average.
The baseline
Sonnet 5, on evidence rather than preference: all 41 of the pipeline's own evaluation fixtures pin it, and its evaluation protocol dispatches candidates on it. It is the model the pipeline was built and tuned against.
Fable 5 is an experiment variable, not a default. It is a premium tier, and most customers will not pay premium prices for a pipeline that runs hundreds of times. Reporting it as the normal configuration would flatter the results and mislead on cost.
What the two columns show
The baseline model is materially slower, roughly twice the wall-clock on the three-repository ticket and two to three times on the two-repository one. Consistent across both, so not one slow stage.
It reviewed more thoroughly in the one case where they differed. On the two-repository ticket it caught two genuine defects where the premium model caught one, the extra find being a new timeout leaking into unrelated calls. One observation on one ticket, not a pattern.
Where they agreed, they agreed completely. Both built the new field in the same non-obvious place. Both judged the resulting mismatch to be added detail rather than a contradiction. Both stopped on the seeded fault at the same point, naming the same repository. Arrived at independently.
Only the baseline column was collected under the fixed measurement boundary, so small differences may be method rather than model. The latency gap is large and consistent enough to survive that. The review difference is not, and is flagged accordingly.
The open question: switching models per stage
A run is not one homogeneous task. Some stages are mechanical: extracting what changed, seeding a record, cross-linking pull requests. Others carry the judgement: planning, implementing, reviewing. One model for everything means either paying premium prices for clerical work or accepting weaker judgement where it matters.
The hypothesis is that mixed assignment beats a single model on both cost and quality. Nothing about it is proven. It is directly testable, because every stage is dispatched separately and can be given its own model.
Strategies become columns in this same table, with the same scenarios and measurements: single baseline, single premium, and mixed assignments. It runs once the scenario suite exists, since comparing on clean runs alone measures only speed and cost, not whether the pipeline still stops in the right places.
The constraint that stays visible
Many clients are locked to one model and cannot act on a per-stage recommendation. That makes the finding a measure of what the lock-in costs, which is worth quantifying. Results stay separated per model so a locked client can read their own row without unpicking an average built from models they cannot use.