Conversation
Nine axes for judging whether an issue can be handed to an agentic LLM system and finished without the finishing act itself creating a maintenance liability. Seven additive (specification closure, oracle strength, blast radius, horizon length, precedent density, context footprint, reversibility), two capping (environmental determinism, design authority). Each axis is anchored to a measured effect rather than intuition: SWE-bench Verified's 68.3% underspecification filter, the verification- horizon result on proxy gaming, the ~2x single-file/multi-file success gap, METR's time-horizon series and the 19%-slowdown RCT, long-horizon per-step compounding, Chroma's context-rot measurements, GitClear's duplication and moved-code trends, and the VibeCheck findings on weak assertions in agent-written tests. Bands are routing instructions (DELEGATE / DELEGATE-WITH-CHECKPOINT / SPECIFY-FIRST / HUMAN-LED / HUMAN-ONLY), not quality judgments, and every grade names the specific debt mode delegation would produce. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuwcuCS5Uv5C8ZRXMXMKdK
… per-comment one A parent is graded before its children's bands exist, so a delegable fraction printed in the parent's own comment would be stale on arrival. Record the declared roster in the grade instead and derive the fraction once, corpus-wide. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuwcuCS5Uv5C8ZRXMXMKdK
…axis definition The first corpus run produced a degenerate DA distribution (87% at exactly 1, none above 3) while every other axis showed spread. Cause was the grading instruction, not the axis: it treated a populated 'Open Questions & Decisions Needed' section as evidence of an open decision, and that section is mandatory in the JLS templates. A 66-issue stratified DA-only re-grade measures the bias at +1.23 mean, strictly one-sided (41 raised, 25 same, 0 lowered). Band impact is 11%, all D->C; nothing enters A or B and the F population is unaffected. The axis definition now excludes evidence hygiene, bookkeeping, questions the issue answers itself via 'Recommended default:', and decisions owned by a different issue. The substantive finding is that JLS's 'Recommended default:' convention closes most of its own open questions and materially raises delegability. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuwcuCS5Uv5C8ZRXMXMKdK
Every open capstone, feature and task graded by an independent agent and the filled rubric posted to the issue. 0 A-band, 8 B-band, 37 C, 421 D, 222 F. Principal findings: ~24% of the backlog cannot be closed by any agent at any capability level because its acceptance evidence requires another person; 58% of predicted debt is one mechanism (executor authors the artifact and the evidence certifying it in the same change); the aggregate delegable fraction of all parent rosters is 8/632. Includes the measured DA-axis defect and its corrected estimate, plus three mechanically fixable findings: 42 issues whose deliverable targets a doc deleted at 4882e93, 194 routing something to the maintainer, and 105 specified against artifacts that do not exist yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuwcuCS5Uv5C8ZRXMXMKdK
Rates how safely an issue can be handed to an agentic LLM system, and
what debt delegating it as-is would create. Delegability is close to
orthogonal to quality: an issue can be excellent work and still band F
because closing it needs an FPGA on a desk. A low band is a routing
decision, not a criticism, and the section says so in every tier.
Seven additive axes (specification closure, oracle strength, blast
radius, horizon length, precedent density, context footprint,
reversibility) and two capping axes (environmental determinism, design
authority), each anchored to a measured effect rather than intuition.
Each tier fills a yaml block and one prose line.
The canonical anchor table lives in scientific_task.md; feature.md and
capstone.md carry only their tier deltas and refer to it, following the
same cross-template convention the edge-legality matrix already uses.
Markdown issue templates have no include mechanism, so the alternative
was three copies that would drift.
Tier deltas: a feature scores its own integration evidence, not the union
of its children, and reports roster_delegable; a capstone is never a
single delegation, so its useful fields are ED on the system-level
acceptance criteria and roster_delegable over its features.
Three further findings from grading the 688 open issues are folded in as
task rules 11-13:
11. Oracle custody -- name who authors the expected values and when.
Same-change authorship of behaviour and golden was the single
largest predicted debt source, 58% of the corpus.
12. Artifact paths resolve at filing. 42 open issues name a deliverable
whose target no longer exists.
13. Open Questions carry an explicit marker: Recommended default:,
BLOCKING:, or HYGIENE:. The templates already asked for a
recommended default in prose; making it a marker is what lets a
reader and the DA axis tell a made decision from an open one.
Without that distinction, grading collapsed 87% of the corpus to
DA=1; with it the axis moved by a measured +1.23.
Verified inert against scripts/validate-issue-template.py on all three
tiers: the new yaml block carries no `tier:` key so it is never mistaken
for the machine block, `band:` does not collide with the banned
`band_mw`, and the new heading shares no leading stem with any canonical
section. Baseline and with-ADR-1 bodies produce identical errors and
warnings.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KuwcuCS5Uv5C8ZRXMXMKdK
Replaces the cross-template reference with three complete copies, so a
filer can score an issue from the one template in front of them. Markdown
issue templates have no include, transclusion or file-reference mechanism
in either the .md or the issue-forms .yml dialect, so single-sourcing
would have required generating the templates from a fragment at build
time. Drift risk is accepted deliberately: when an axis changes, it must
change in all three.
The copies are re-anchored per tier rather than duplicated verbatim,
because the task anchors are meaningless at the other two tiers -- "one
file, or one new file plus its test" does not describe a capstone's blast
radius.
task unchanged; scores the change itself.
feature scores the feature's OWN deliverable, the integration evidence
in section 5 and the contract in section 3, never the union of
its children. BR/HL/CF are re-anchored to the
wait-then-reconcile-then-verify chain and are 0-1 for a
composing feature by construction. OS calls out that "every
child closed" is the roster, not an oracle. Adds
roster_delegable over requires_tasks.
capstone scores section 4's system-level acceptance criteria. HL and CF
are stated as 0-1 by construction with the honest two-value
anchor; a score of 2-5 means this is not a capstone. RD notes
that capstones are where irreversible-outside-the-repo
concentrates, because capstones are what announce things. ED
is marked the key field of the block, since it is invariant to
model capability. Adds roster_delegable over
requires_features.
Each copy now carries the full nine anchors, both deductions, the band
arithmetic, the debt-tag precedence, and the three filing checks the
scientific-task template states as rules 11-13.
Re-verified against scripts/validate-issue-template.py: synthetic
conforming bodies for all three tiers produce identical errors and
warnings with and without the section.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KuwcuCS5Uv5C8ZRXMXMKdK
The sections had specialized twice over: to this project's toolchain and file formats, and to the one grading exercise that produced them. Both kinds of detail rot, and neither belongs in a template a filer reads years from now. Every domain-specific name and every corpus statistic is gone; the anchors are now written against generic software concepts -- a published format, a registry, an external toolchain, a repository-wide invariant, the project's standard build-and-test command. A short rationale block replaces the statistics with the general effects the axes are anchored to, stated as effects rather than as numbers from one backlog. Six defects found by reviewing the generated result rather than the generator: 1. GOLDEN-LOCK-IN contradicted its own trigger. It fires on RD=2 AND OS>=4, but it was described as an expected-output artifact authored by the work it certifies -- which fires the OS deduction, capping OS at 3, so the trigger could never fire for the case it described. Re-described to match: a load-bearing expected artifact later work is graded against, correct today, whose liability is entrenchment. The same-change case is HOLLOW-ORACLE, and the boundary is now spelled out in both tags. 2. Tags forced by a tier's structural bounds carried no information. PARTIAL-INTEGRATION fires on CF<=1 and SCOPE-EXHAUSTION on HL<=1, both of which the outcome tier declares bounded by construction, so both were guaranteed there and said nothing about the issue. Added the rule that a tag met only because of a tier bound reports the tier, not the issue. 3. Band A is unreachable at the outcome tier -- with two axes bounded at 1, RAW cannot exceed 27 -- and nothing said so, leaving a filer to wonder why the top band never occurs. Now stated as intentional. 4. That tier's action enum omitted DELEGATE-WITH-CHECKPOINT even though band B is reachable at RAW 24-27. 5. "0-1 by construction" overstated the composition tier's horizon: a thin feature really can be a one-hour integration check. Softened to the ordinary case, with a prompt to justify a high score. 6. The version marker had no bump protocol, so drift between the three copies was undetectable by the one mechanism provided for it. Changing an axis now requires bumping the marker in every copy; a copy whose marker disagrees with its siblings is stale by definition. Verified: no residual project vocabulary in any section, all nine axes and all eight debt tags present in each, no ADR yaml block carries a `tier:` key, and synthetic conforming bodies for all three tiers still produce identical validator errors and warnings with and without the section. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuwcuCS5Uv5C8ZRXMXMKdK
Band A was unreachable there, but not because anything said so: two axes were bounded at 1, which caps RAW at 27 against a threshold of 30. That is a numerical side effect standing in for a design statement, and it was wrong twice over. It contradicted the section directly above it. The header says to score this tier's OWN deliverable and never the union of its features; the horizon anchor then forced the score to reflect the composed multi-week programme, which is exactly that union. The composition tier already had the correct rule -- measured after the roster lands, not the sum of the children's horizons -- and it simply was not applied one tier up. It also made the rubric lie about real issues. A capstone whose features have all landed and whose acceptance is one scripted walk-through against a committed transcript has a short chain and a small footprint, and must be allowed to score like one. Waiting on an unlanded roster is an ordering dependency: it belongs in the roster and in blocked_by, not in a horizon score. Horizon Length and Context Footprint at that tier now carry full 0-5 anchors scoped to the acceptance pass -- demonstrate, reconcile, record -- with an explicit note that scoring high while the roster is incomplete means the wrong thing was scored. Blast Radius likewise: acceptance that only observes scores high, acceptance that must change production code to become demonstrable scores low, and that is a signal the features did not finish the job. Every band is now reachable at every tier. The tier still tends to land low, because acceptance criteria are the least likely to be closed and the most likely to need a person -- but that now has to be earned by the scores rather than imposed by the arithmetic. The action enum regains DELEGATE, and the yaml axis comments no longer assert bounds they should not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuwcuCS5Uv5C8ZRXMXMKdK
Nine reviewers -- three lenses (internal consistency, adversarial filer, generality and rot) across all three templates, none told who wrote the section -- returned 137 findings, 29 critical, and all nine verdicts were "not fit to ship". The convergent findings were right. Design Authority contradicted itself in three places at once. Anchor 3 scored a named decision carrying a stated preference as 3 (cap B) while the exclusion list said that exact case does not lower the axis at all; five reviewers found it independently. Since the templates instruct filers to give every open question a recommended default, the anchor reading capped essentially every conforming issue at B, and the section's own advice -- that answering an open question is the cheapest way to raise a band -- was false. Anchor 3 is now for a preference offered pending someone else's confirmation; a preference an executor may act on is a decision MADE and scores 4. The axis also routed decisions owned by another issue to Horizon Length, which has no anchor that can receive them and which now explicitly refuses waiting; the case was scored nowhere. An unmade decision owned elsewhere is anchor 1. Only an ANSWERED decision cited from elsewhere is excluded. Both capping axes were monotone in disclosure: every penalty keyed on what the filer wrote down, so an issue that named no decisions and demanded no awkward evidence scored best. That is the exact inverse of the intent. Both now say to score what the work requires, not what the text lists, and an "N/A -- fully specified" no longer supports a high score by itself. Environmental Determinism banded F on "a second maintainer holding approval rights" -- ordinary branch protection in most projects, and the literal definition of band B's checkpoint. It now scores only people whose participation produces evidence; review and merge approval are explicitly not ED factors. Oracle Strength gave 4 for any standing project-wide gate, which every change already passes, so the axis was pre-inflated on every code-touching issue. It now requires a gate this change would fail before the fix. The tag boundary fixed last commit only held for the 2-point deduction; the 1-point deduction left both GOLDEN-LOCK-IN and HOLLOW-ORACLE firing at once. HOLLOW-ORACLE is now keyed to the 2-point deduction specifically. The yaml also could not record which deduction fired, though two tags depend on it; added os_deduction. At the outcome tier, the Horizon Length 2-anchor restated the tier's own entry condition -- criteria no single feature covers -- so every legal issue of that tier matched it and the axis was pinned low by a second route. Removed. Its two horizon instructions also contradicted each other on whether an incomplete roster forbids a high score; the rule is now to score the pass as it will be once the roster lands and never to count waiting. And the acceptance pass could be emptied on paper by assigning its harness to a feature; unbuilt machinery is now scored wherever the work sits. Rejected one finding: tier-adjusted band thresholds. It was written against wording that called two axes structurally bounded, which the previous commit removed; with every axis unbounded at every tier the shared 0-35 thresholds are correct. Verified: no residual project vocabulary, all nine axes and eight tags present in each copy, and synthetic conforming bodies still produce identical validator diagnostics with and without the section. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuwcuCS5Uv5C8ZRXMXMKdK
Round 2 of independent adversarial review -- 15 agents, five lenses, blind
to round 1 -- returned 230 findings and 15 of 15 "not fit to ship". The
number that mattered was not the count but the trend: defect density per
agent was flat (15.2 -> 15.3 findings, 3.2 -> 2.9 critical), and 80% of
the new criticals quoted wording introduced by the round-1 FIX commit.
Patching was creating defects about as fast as it removed them, because
every clarification added prose and prose was the substrate the
contradictions lived in.
The structural cause was two places to score one fact. Each capping axis
had an anchor ladder AND a parallel exclusion list, and four of the top
clusters were the two disagreeing: an unmarked open question scored 2 by
the filing check and 5 by the hygiene exclusion and 1 by the BLOCKING
marker; a decision owned elsewhere was anchor 1 and simultaneously
excluded; a self-answered question was anchor 3 and simultaneously
excluded.
v2 folds every exclusion into the ladder it contradicted, so each fact
pattern is scored in exactly one place, and states that rule explicitly.
The sections drop from ~300 lines to ~95 of rubric, and from 61-65% of
each template to 34-50%.
Substantive fixes carried in the rewrite:
- DA anchors now name the marker that produces them, and rule 13 gains
the fourth marker the ladder needed: PROPOSED, a preference the filer
may draft but not land. Anchor 3 previously described a state the
marker vocabulary could not express.
- DA anchor 4 no longer self-certifies: a stated preference clears the
cap only when forced by existing code, by a declared contract, or by
the filer's own authority.
- An unmarked open question is scored on its substance and the marker
fixed, rather than scored worse than the same entry marked honestly,
which had inverted the incentive the marker rule exists to create.
- ED distinguishes reviewing a diff, which is the band-B checkpoint and
does not score, from running the software and reporting what happened,
which is evidence production and does.
- OS deductions are cumulative with a stated floor; os_deduction records
the total 0-3, since both can fire at once and a 0|1|2 field could not
say so.
- New tag IRREVERSIBLE-PUBLICATION for RD<=1, which no tag covered.
- Debt tags are selected by first trigger in a fixed order, replacing
"most severe applicable", which required a severity judgment the list
never supplied.
- roster_delegable takes `pending` when children are unscored, which is
the common case at filing.
Verified: all nine axes and all nine tags present in each copy, no
residual project vocabulary, no parallel exclusion list remaining, and
synthetic conforming bodies still produce identical validator diagnostics
with and without the section.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KuwcuCS5Uv5C8ZRXMXMKdK
Two checkboxes and a machine field recording whether an adversarial read and a peer read of the issue itself left anything substantial outstanding. This is the cheap filter the scores could not provide on their own: an ADR-1 band describes the issue as written, and says nothing about whether anyone has tried to break it. Self-ticked by design, per the maintainer's call: the implementer is assumed competent, so a separate reviewer is not required and the gate stays fast. "Substantial" is defined so the box means the same thing across filers -- a finding is substantial when acting on it would change a score, a completion criterion, a prediction, or the scope; wording is not. Unticked means not yet reviewed, which is neither a defect nor a bar to filing; it means the band is still unvalidated. review_evidence points at a review comment where one exists, and is optional. Placement checked rather than assumed: the boxes sit in the delegability section, and validate_hygiene's H09 scopes its ticked-box audit to section_content(DOD_HEADING[tier]), so these cannot be mistaken for Definition-of-Done items. The validator remains inert -- synthetic conforming bodies produce identical diagnostics with and without the section. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuwcuCS5Uv5C8ZRXMXMKdK
Eight of fifteen independent reviewers cited one sentence added in the v2 cut: 'where two rules could apply to one fact, the anchor ladder wins'. Read literally it nullifies the specification-closure cap and both oracle deductions, since every non-anchor rule in the section speaks to facts an anchor also speaks to. It was introduced as the fix for the contradiction problem and became the document's most-cited defect. Scoped to anchor-vs- anchor, with caps and deductions applying on top. Also: GOLDEN-LOCK-IN could not fire. Its trigger required OS>=4, whose anchors describe comparison against a PRE-EXISTING artifact, while RD=2 describes this work COMMITTING the load-bearing one -- the two could not hold together. Trigger reduced to RD=2; the tag ladder's first-match order already routes self-certified cases to HOLLOW-ORACLE above it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuwcuCS5Uv5C8ZRXMXMKdK
The previous commit landed them on the task template only: the preamble wraps differently in the other two copies and the replacement did not match, which is the drift cost of three standalone copies arriving on schedule. All three now carry the same rule. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuwcuCS5Uv5C8ZRXMXMKdK
A control run settled how these review numbers should be read. The same twelve agents, same four lenses, same instructions, pointed at the mature pre-existing templates -- written by the maintainer, iterated for months, governing 688 open issues -- returned 13.9 findings and 2.4 criticals per agent, and twelve of twelve "not fit to ship". The new section reads 16.6 and 3.0, also twelve of twelve. A ratio of 1.19. The instrument does not discriminate. Three rounds of density measurement said almost nothing about this section's quality, and the reasoning built on that trend -- that patching was not converging, and that length was the cause -- rested on an uncalibrated measurement. That is the same error as the design-authority axis, one level up, and it ran for three rounds instead of one. What does discriminate is convergence: how many independent reviewers quote the same text. Thirteen of fifteen cited the tie-break sentence, and it was a real defect. One of fifteen cited an undefined term, and it was not. Applying a three-reviewer bar to round three leaves three findings, all fixed here: ED anchors 2, 1 and 0 overlapped (9/15): a platform-dependent test matched all three, so the same evidence capped at C or at F depending on the reader. They are now disjoint on a single question -- could automation ever produce this unattended -- answered yes-but-not- provisioned, no-because-physical, no-because-human. DA anchors 3 and 1 both claimed a PROPOSED entry (7/15). Split on whether work can begin: BLOCKING stops a start and is 1, PROPOSED lets work proceed against the stated preference and is 3. roster_delegable ignored unfiled planned entries and, at the outcome tier, requires_capstones (3/15). Now defined over every filed composition entry, with unfiled planned entries making the fraction provisional. Findings below the bar are left alone deliberately. On this evidence they are indistinguishable from what the same lenses produce against a document already in production. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KuwcuCS5Uv5C8ZRXMXMKdK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nine axes for judging whether an issue can be handed to an agentic LLM
system and finished without the finishing act itself creating a
maintenance liability. Seven additive (specification closure, oracle
strength, blast radius, horizon length, precedent density, context
footprint, reversibility), two capping (environmental determinism,
design authority).
Each axis is anchored to a measured effect rather than intuition:
SWE-bench Verified's 68.3% underspecification filter, the verification-
horizon result on proxy gaming, the ~2x single-file/multi-file success
gap, METR's time-horizon series and the 19%-slowdown RCT, long-horizon
per-step compounding, Chroma's context-rot measurements, GitClear's
duplication and moved-code trends, and the VibeCheck findings on weak
assertions in agent-written tests.
Bands are routing instructions (DELEGATE / DELEGATE-WITH-CHECKPOINT /
SPECIFY-FIRST / HUMAN-LED / HUMAN-ONLY), not quality judgments, and
every grade names the specific debt mode delegation would produce.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01KuwcuCS5Uv5C8ZRXMXMKdK