Skip to content

The report of the second campaign of the evaluations, kept - #84

Merged
PierreMardon merged 1 commit into
mainfrom
docs/campaign-2-report
Oct 3, 2026
Merged

PierreMardon merged 1 commit into
mainfrom
docs/campaign-2-report

Conversation

@PierreMardon

Copy link
Copy Markdown
Contributor

The report of the second campaign of the evaluations, kept for the next one to be compared with: evals/reports/2026-10-03-e830bc236e9b/. It measures the chain after the four first fixes drawn from the first campaign (#78, #79, #80, #81), on 01-overdue-list and 03-overdue-reminders, three runs each: 67 sessions, 17.07 USD at list price.

What was done

  • report.md and summary.json of the campaign, as report --against evals/reports/2026-10-03-6bf7c06609a6/summary.json --keep wrote them. Nothing else changes.

What it shows

  • Every run reached conformant and passed all of its hidden acceptance tests.
  • What moved since the first campaign, by the report's own rule: on 01-overdue-list the interview asks one question where it asked none, the order of ties, in three runs of three; on 03-overdue-reminders the cut of the blueprint and the messages score one point better.
  • What did not: blueprint.no-padding is 2 on every run, the extractor being unchanged; following.no-noise is 2; the developer still sends one blueprint back per run on average.
  • First spreads between runs of one case: a run of 01-overdue-list costs from 1.41 to 3.60 USD and takes from none to two corrections.

The conclusions, issue by issue, are in the issues #77 lists.

Tests

No code changes. The text rules of the repository pass on the two files; the CI runs every gate.

Refs #77

The report of the second campaign of the evaluations, kept for the next one to be compared
with: `evals/reports/2026-10-03-e830bc236e9b/`. It measures the chain after the four first
fixes drawn from the first campaign (#78, #79, #80, #81), on `01-overdue-list` and
`03-overdue-reminders`, three runs each: 67 sessions, 17.07 USD at list price.

Refs #77
@PierreMardon
PierreMardon merged commit 70a5c4d into main Oct 3, 2026
3 checks passed
@PierreMardon
PierreMardon deleted the docs/campaign-2-report branch October 3, 2026 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant