Skip to content

feat(scenarios): the scenario scorer moves into @objectstack/scenarios and judges capability tags, agent-action exposure, write outcomes and per-persona answers - #30594

Merged
objectstack-fleet[bot] merged 11 commits into
mainfrom
claude/issue-30291-scenario-tag-checks
Nov 5, 2026
Merged

objectstack-fleet[bot] merged 11 commits into
mainfrom
claude/issue-30291-scenario-tag-checks

Conversation

@objectstack-fleet

@objectstack-fleet objectstack-fleet Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #30291
Clause-②: no
Metadata-protocol: none

维护者速读

  • 改了什么:确定性场景评分器整体从 packages/cli 迁入私有包 @objectstack/scenarios(裁决 D)。新增封闭的能力标签检查表、Agent 动作暴露检查、写操作结果判分和按角色判分,并提供给运行器用的入口 @objectstack/scenarios/score。
  • 为什么改:维护者裁定方案 D(「同意 D」),评分器和卡片格式放在一起。已发布的 CLI 不再携带任何场景代码,也不再需要依赖私有包。
  • 风险与回滚:CLI 只把 MetadataEvalCase 类型挪到 corpus.ts 并原名重导出,行为不变。评分器此前没有从 CLI 发布过。回滚即撤销本 PR。
  • 先看:packages/qa/scenarios/src/score/scenario-tags.ts(标签表与动作暴露判定)、packages/qa/scenarios/src/score/index.ts(运行器入口)、.changeset/30290-scenario-scorer-port.md 的删除(见下方"需要你做的")。
  • 两分钟验证:pnpm --filter @objectstack/scenarios test,预期 11 个文件、163 个用例全部通过。
  • 需要你做的:确认删除尚未发布的 .changeset/30290-scenario-scorer-port.md。它写的是"CLI 新增评分器",迁移后这句话已不成立。check-empty-changeset 对这类有意更正会一直保持红色,直到有人确认。

What changes

Per the maintainer's ruling D on #30291 (「同意 D」), the deterministic scenario scorer lives in the private @objectstack/scenarios package (packages/qa/scenarios), beside the card format it judges. No model call anywhere (cost ruling on objectstack-ai/cloud#3600: 「目前阶段尽量不要用云端的token,api的更贵」).

  • Moved with git mv, so history follows: scenario-score, scenario-graph and scenario-run-grade (merged in feat(cli): port the deterministic scenario scorer — shape score, artifact graph reader and Run grader — with a discrimination self-test #30332), scenario-tags and scenario-operation-grade, and their tests. They go from packages/cli/src/lint/ to packages/qa/scenarios/src/score/ and test/score/. @objectstack/cli ends with no scenario code and no mirrored card types.

  • The graders take the card schema's own types: ScenarioExpect, z.infer of OutcomeSchema and PostStateSchema, a Pick of ScenarioCard, and the loader's Truth. The expect, outcome, persona, agent-action and truth mirrors are gone.

  • @objectstack/scenarios/score (private subpath) is the runner's entry (Scenario library T3/T4: local Claude Code runner — Build, Run (问数 + operations per persona over the app's MCP), Iterate — writing run records, zero cloud-billed tokens #30292). It has three functions, and each takes a card the loader returned:

    • scoreCardArtifact(card, artifact);
    • gradeCardOperation(card, operationId, observation);
    • gradeCardAnswer(card, { question, persona?, turn }). The persona's hidden objects and fields are read off the card.
  • Capability-tag checks (scenario-tags.ts): a closed table of nine rows, listed in the package README for the mid-tier card authors.

    • A tag with no row is unscored, never a pass. So is a row whose card gives it nothing to judge. Both are counted nowhere.
    • A row that runs zero checks fails.
    • Rows read the owning predicates (resolveFlowTriggerKind, APPROVAL_NODE_TYPE, SINGLE_OPTION_TYPES, summaryFunctionReadsField, effectiveSharingModel, standaloneActionOwnerKey). Collections are read through resolveStackCollection from its owner, @objectstack/core.
  • Agent-action exposure: each declared action must exist on its object, carry ai.exposed: true (the MCP doors' opt-in) and, where the card says confirm, carry ai.requiresConfirmation: true (the flag the confirmation gate enforces, not the delete-mode listing heuristic). A missing action fails all of its checks.

  • Write-operation grader:

    • applied: every post-state is read back, one row per where, and the audit actor matches;
    • refused: the exact ErrorCode, with zero writes;
    • pending_confirmation: zero writes.

    An unobserved fact fails.

  • Per-persona Run grading:

    • a withheld truth is must-not-reveal: a decline is correct, and a stated all-rows figure is a leak;
    • an all-rows figure outside a scoped persona's rows is a leak when the reply states it.
  • Dependencies of the private package: @objectstack/core, @objectstack/objectql, @objectstack/plugin-sharing (workspace). Its vitest config aliases each to source (check:test-source-alias).

  • Gates re-homed:

    • the cross-package inputs move from @objectstack/cli#test to the scenarios entry and task, which also declare packages/cli/src/lint/corpus.ts (the golden builds and the drift pin read it);
    • the second-object-read-exposure pin row for scenario-graph.ts follows the file (same not-a-served-read class).
  • CLI: MetadataEvalCase is declared in corpus.ts, beside the corpus it types, and metadata-eval.ts re-exports it under the same name. That keeps the scenarios type-check program from pulling in the whole lint command.

  • Changesets: one changeset (@objectstack/scenarios and @objectstack/cli, patch). The unreleased .changeset/30290-scenario-scorer-port.md is deleted on purpose: it announces a scorer in the CLI that this PR moves out, so it is a deliberate correction of a pending release note. check-empty-changeset stays red on it until a person confirms.

  • Body item 2 (os lint --eval reads cards) is cancelled by the ruling.

Tests

  • pnpm --filter @objectstack/scenarios test: 11 files, 163 tests. Moved suites unchanged in what they assert. Added test/score/entry.test.ts (the runner entry over the real crm_account_with_contacts card).
  • scenario-cards.test.ts loads the control cards through the loader:
    • corpus drift pin. Each control card's expect must equal what corpus.ts derives, with a floor of exactly 3 controls;
    • end-to-end readings: crm 10/18, invoice 18/27, project 19/26;
    • crm completed with what it asks scores 1, and each removal scores lower.
  • pnpm --filter @objectstack/scenarios typecheck: exit 0.
  • pnpm --filter @objectstack/core exec vitest run src/security/second-object-read-exposure.pin.test.ts: 4/4.
  • pnpm --filter @objectstack/cli typecheck, build, and vitest run test/metadata-eval.test.ts src/lint/: 43/43.

Acceptance notes

Verification

One ablation per moved module, at the new home (scripts/ablation-replace.mjs; tests import src relatively, so no dist leg)

Every mutation landed (anchor 1 → 0) and was restored (blob == HEAD, git diff HEAD empty):

module mutation result
scenario-score widget measure check → pass 2 failed / 16
scenario-graph nav entry resolves → true 2 failed / 5
scenario-run-grade withheld leak scan off 1 failed / 19
scenario-tags master-detail inlineEdit → true 1 failed / 30
scenario-operation-grade no-write → true 2 failed / 13
score/index operation judged against the wrong outcome 1 failed / 7

The full per-check battery of the previous round (18 mutations) ran against the same assertions before the move.

Gates

  • node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands on the patch-round diff: 91 commands, all run, exit codes captured before any pipe.
  • check-empty-changeset --base origin/main: exit 1, the deliberate-correction refusal on 30290-scenario-scorer-port.md (above).
  • check:dual-build-cjs-loads and check:i18n-coverage: exit 3 PREREQUISITE NOT MET. They read every package's built output; NOT MEASURED, CI owns them.
  • Every other family: exit 0, including check:cross-package-test-inputs, check:ci-filter-parity, check:turbo-task-graph, check:undeclared-dep-imports, check:test-source-alias and check:workspace-manifest-cycles.
  • Clause-② closure: walking the relative imports of the CLI's dist/index.d.ts, dist/console.d.ts and dist/hook-body.d.ts reaches 39 files, none a scenario file, and no bare @objectstack/scenarios specifier. dist/lint/ holds only corpus, hook-body-lowering, metadata-eval and score.
  • Dogfood: n/a. No ! in the title and no (narrowing) arm.

Generated by Claude Code

An ablation showed record-triggered-flow-exists could never decide a verdict
on its own: a build with no such flow already fails the binding check.

Claude-Session: https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added size/xl documentation Improvements or additions to documentation tests tooling Fix lands in tooling, not a product package — enters pm:queue only with Unblocks: or a named surface labels Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/cli, @objectstack/scenarios, touching 95 documentable anchor(s). ⚠️ 3 changed file(s) yielded no anchor (packages/qa/scenarios/README.md, packages/qa/scenarios/package.json, packages/qa/scenarios/vitest.config.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

39 hand-written doc(s) name something this change touched — list omitted above 15 rows. Re-derive on the tree named below: node scripts/docs-audit/affected-docs.mjs --json a2437b560bd8872feece6ec8b57c6fdf21cff1f4.

⛔ 13 release-owned page(s) also affected — read-only, see AGENTS.md Documentation Guardrails.

What this run could not see
  • 3 changed file(s) yielded no anchor (packages/qa/scenarios/README.md, packages/qa/scenarios/package.json, packages/qa/scenarios/vitest.config.ts) — pages documenting those are invisible to this run
  • 2 anchor(s) matched too much of the corpus to be a work list: master_detail (literal, 40 pages), objectName (literal, 56 pages)
  • 53 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 63 of 221 client-bound route-ledger rows — the other 158 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 158: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 103 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 35 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json a2437b560bd8872feece6ec8b57c6fdf21cff1f4 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from cf781c2383720f9402a6355dac14b1692ef81b70 — the merge of head 4dd3edc52bdfe115b3b86dce74a3b31df4450e4a into base a2437b560bd8872feece6ec8b57c6fdf21cff1f4, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin cf781c2383720f9402a6355dac14b1692ef81b70 && git checkout cf781c2383720f9402a6355dac14b1692ef81b70
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin a2437b560bd8872feece6ec8b57c6fdf21cff1f4 4dd3edc52bdfe115b3b86dce74a3b31df4450e4a && git checkout -B drift-repro a2437b560bd8872feece6ec8b57c6fdf21cff1f4 && git merge --no-ff 4dd3edc52bdfe115b3b86dce74a3b31df4450e4a

node scripts/docs-audit/affected-docs.mjs --json a2437b560bd8872feece6ec8b57c6fdf21cff1f4

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs a2437b560bd8872feece6ec8b57c6fdf21cff1f4 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

…int into @objectstack/scenarios

Pure rename, so history follows the files; imports are rewired in the next commit.

Claude-Session: https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu
Co-authored-by: Claude <noreply@anthropic.com>
…eset for the moved scorer

The unreleased CLI changeset for the scorer port is folded in: the modules it
described no longer ship from @objectstack/cli.

Claude-Session: https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu
Co-authored-by: Claude <noreply@anthropic.com>
@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

Contract review — PR #30594

Head-sha: 4dd3edc
Verdict: PASS

This is the record of an isolated, read-only contract-tier reviewer the epic seat ran on this head (the seat posts it; the reviewer wrote nothing).

  1. The move preserves behaviour. The move commit e4279a2d is pure renames: 11 files, 0 insertions and 0 deletions. After it, the source changes are rewiring only:

    • real schema types replace the mirrors (ScenarioExpect, z.infer of OutcomeSchema / PostStateSchema, Truth, a Pick of ScenarioCard);
    • resolveStackCollection comes from its owner @objectstack/core;
    • measureChecks is exported.

    No check body changed, and no assertion was weakened.

  2. CLI cleanup. No scenario code and no mirrored types remain under packages/cli. packages/cli/package.json is unchanged. The MetadataEvalCase move into corpus.ts is interface / import type / export type only, so it emits no JS.

  3. Clause-② no. There is no new @objectstack/cli entry export. @objectstack/scenarios stays private: true, and its ./score subpath is a private-package entry. src/index.ts does not re-export it.

  4. Security pin. The second-object-read-exposure row is re-keyed to packages/qa/scenarios/src/score/scenario-graph.ts, with the same not-a-served-read class and the same reason, which is still true: it reads the artifact only, with no record, fetch or child_process.

  5. The changeset correction holds, judged sentence by sentence (table below).

  6. CI on this head: 70 check-runs. The only failures are Check Changeset ×2, the designed DELIBERATE CORRECTION red.

# Deleted .changeset/30290-scenario-scorer-port.md (pending, unreleased) False after this PR?
O1 '@objectstack/cli': patch as the scorer's bump yes: the CLI no longer ships the scorer
O2 "feat(cli): a deterministic scenario scorer under src/lint …" yes: nothing scenario-related remains under packages/cli/src/lint
O3/O4 Clause-②: no / Metadata-protocol: none still true; carried verbatim in the new note
O5/O7/O8 Structural scorer / artifact reader / Run grader at src/lint/scenario-*.ts yes as CLI notes: those paths are gone; the functions are unchanged in the moved modules
O6 The two closed shape checks yes as a CLI statement; the behaviour survives in the private package, never released
O9 "… the os lint --eval wiring is a later change" yes: cancelled by ruling D (comment 6670490510 on #30291)
# New .changeset/30291-scenario-tag-checks.md True?
N1 @objectstack/scenarios and @objectstack/cli patch yes
N2/N5–N7 the scorer lives in @objectstack/scenarios; the private ./score entry (scoreCardArtifact, gradeCardOperation, gradeCardAnswer) takes loader-returned cards and uses the schema's own types yes
N8 "moved here from @objectstack/cli … never released from the CLI" yes: pure-rename commit; no scorer entry in the CLI CHANGELOG
N9–N13 a closed 9-tag table (unscored counted nowhere), agent-action exposure (ai.exposed, ai.requiresConfirmation), the write-outcome grader, per-persona grading yes
N14–N16 the CLI's MetadataEvalCase relocation; no behaviour change, flag or export; no model call yes

The deletion loses no true user-facing statement: every still-true fact of the old note is restated in the new one.

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

席位核对

Verdict: ACCEPT at head 4dd3edc52bdfe115b3b86dce74a3b31df4450e4a.

  1. Shape.
    • Line 1 is Fixes #30291. Body item 2 (os lint --eval reads cards) is cancelled by maintainer ruling D, verbatim 「同意 D」 (Scenario library T2b: capability-tag checks in the ported scorer + os lint --eval reads scenario cards #30291, comment 6670490510).
    • Clause-②: no / Metadata-protocol: none sit at line start in the body and the changeset.
    • The body was written by the seat from the dev's replacement (comment 6671781042).
    • The title is corrected in this stroke from feat(cli) to feat(scenarios): the reviewer flagged that the CLI no longer ships the scorer.
  2. Size. PR additions + deletions = 1806 (+1741/−65), the SKILL.md authority, under 3000. dispatch-gates reads 4324 because it counts with --no-renames; that tool defect is filed as dispatch-gates / check-governed-merges count changed lines with --no-renames, so a rename-heavy PR reads far over the 3000-line threshold that GitHub's additions+deletions (the authority) puts it under #30718.
  3. Scope. Inside the surface amended by the ruling, plus gate-forced files: vitest.config.ts source aliases, pnpm-lock.yaml, and the type-only corpus.ts / metadata-eval.ts move that keeps the CLI's lint command out of the scenarios type-check program. All are accepted deviations. No packages/spec change; no public surface.
  4. Tests and gates.
    • Dev side: 11 files / 163 tests in @objectstack/scenarios; the core security pin 4/4; CLI typecheck, build and lint tests 43/43; one ablation per moved module, each red then restored.
    • CI on this head: everything green except Check Changeset (see 5).
  5. DELIBERATE CORRECTION, confirmed (landing-operations: for a PR that also changes code, a same-head contract-tier review PASS that judges the rewritten note sentence by sentence). The note is .changeset/30290-scenario-scorer-port.md, pending and unreleased, from merged PR feat(cli): port the deterministic scenario scorer — shape score, artifact graph reader and Run grader — with a discrimination self-test #30332. What changed under it: this PR moves the scorer it announces out of @objectstack/cli. The judgement is the contract review above (PASS, same head). ⛔ The note is not restored: restoring it would publish a now-false CLI changelog line.
  6. Queueing with the designed red: all three conditions of SKILL.md hold.
    1. The gate's own source says it is red by design on a pushed branch for this class: scripts/check-empty-changeset.mjs — "this gate stays red either way, and staying red is what puts the decision in front of a person".
    2. Check Changeset runs in .github/workflows/pr-automation.yml, which triggers on pull_request only, never merge_group.
    3. This comment records the gate and the reason.
  7. Landing. Ready, then auto-merge or merge queue through the relay in the next stroke.
  8. Out-of-scope findings.

@objectstack-fleet objectstack-fleet Bot changed the title feat(cli): the scenario scorer judges capability tags, agent-action exposure, write outcomes and per-persona answers feat(scenarios): the scenario scorer moves into @objectstack/scenarios and judges capability tags, agent-action exposure, write outcomes and per-persona answers Nov 5, 2026
@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review November 5, 2026 23:13
@objectstack-fleet
objectstack-fleet Bot enabled auto-merge November 5, 2026 23:13
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Nov 5, 2026
Merged via the queue into main with commit 6ef7f58 Nov 5, 2026
70 of 73 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-30291-scenario-tag-checks branch November 5, 2026 23:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation size/xl tests tooling Fix lands in tooling, not a product package — enters pm:queue only with Unblocks: or a named surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scenario library T2b: capability-tag checks in the ported scorer + os lint --eval reads scenario cards

1 participant