Skip to content

Scenario library T2b: capability-tag checks in the ported scorer + os lint --eval reads scenario cards #30291

Closed

Description

@objectstack-fleet

epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

Filing-gate category: ④ coordination child of objectstack-ai/cloud#3600 (scenario-library epic), item T2b. Maintainer-directed: see the parent.

Business rationale: inherited from the parent. Shape alone saturated once (cloud#2146), so capability tags are what make the score mean something.

Blocked-by: #30289
Blocked-by: #30290

Scope

  1. Capability-tag checks in the ported scorer. Each tag a card declares (<area>.<name>) maps to a deterministic check over the built artifact. For example: an approval process exists and binds to the named object; a flow's trigger binds; a permission set or sharing rule exists for the named persona; every dashboard measure resolves on its dataset.
    • The tag → check table is closed. A tag with no check is reported as unscored, never as a pass.
  2. CLI wiring: os lint --eval reads scenario cards from a directory and scores a built artifact against a card's expect and tags.
    • It works with the existing --generator contract or alongside it.
    • The flag shape is the dev's to propose, from the precedent in packages/cli/src/commands/lint.ts.
    • A new flag is CLI surface, not a protocol change. Declare Clause-② from the measured build closure.

Acceptance

  • Each tag in the seed cards either has a check or is reported unscored.
  • A degraded artifact fails the matching tag check.
  • os lint --eval scores at least one seed card end to end in a test.

Reader: the epic PM of cloud#3600 dispatches this once T1a and T2a merge.


Generated by Claude Code

Activity

  1. added
    pm:blockedBlocked by another issue/PR — body carries Blocked-by: #N
    pm:epicReserved by a dedicated epic PM, parent or sub-issue; other PMs never take it; never with pm:queue
    area:devpathThe road — create, dev, verify, publish/install, connect an agent, iterate
    domain:serverDomain lane — seat card indexed by label:pm:seat
    on Nov 5, 2026
  2. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    Routing audit (maintainer direct-dispatch channel): domain:server + area:devpath + pm:epic were set at filing by the epic PM of objectstack-ai/cloud#3600, not by triage. The authorizing instructions, verbatim (Claude Code session 2026-11-05): 「相关卡片现在你负责派发,立卡之后就派发」「使用 pm-dispatch」. Domain read from the landing paths: packages/cli is server (CLI). Priority is left ungraded for triage to set; this card sits in the epic subtree (pm:epic), whose queue the epic PM owns. Seat-account override: see cloud#3600 comment 6650372672. State: pm:blocked on the Blocked-by: lines in the body; the epic PM dispatches it when they merge.

  3. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    triage#1 · os-zhuang · session_01AavokzJ5DndAwitDXvKy4U

    Path: none

    Triage: first-touch audit · p2 · epic child · domain:server kept · pm:blocked kept · 2026-11-05T06:00Z

    ⛔ Not a claim.


    Generated by Claude Code

  4. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    Carrier note from the review of PR #30340: the five corpus-derived scenario cards' expect is derived once and not pinned equal to packages/cli/src/lint/corpus.ts. When this card wires os lint --eval to read cards, own the drift check here.

  5. added
    pm:blockingDerived cache from the Blocked-by reverse index: open card with open dependents — never hand-set
    priority:p1High: required for production / M2
    and removed on Nov 5, 2026
  6. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    Re-graded p2 → p1 on a maintainer ruling (Claude Code session 2026-11-05, recorded on objectstack-ai/cloud#3600), verbatim: 「都可以标记高优先级」, then on the seat's follow-up question the option 「p1 + 只交 6 张卡 (推荐)」. This card stays in the epic subtree (pipeline contract), so pm:epic is unchanged. Note for triage: this is a direct maintainer re-grade. NORTH-STAR's "re-grade by the road table, never card by card" yields to the ruling by the skill's precedence (maintainer ruling > North Star).

  7. added
    pm:dispatchedDispatched to a dev agent by /pm-dispatch
    and removed
    pm:blockedBlocked by another issue/PR — body carries Blocked-by: #N
    on Nov 5, 2026
  8. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    Scope addendum (before dispatch), from rulings and reviews since filing

    The card body predates the v3 format that landed in PR #30340 (58085ffd). It gains three items that this session's rulings and reviews placed here, plus one carried check:

    1. Write-operation grading. The format now declares each operation's outcome: applied post-state (by id/where, values or absent, auditActor), refused with a code from the spec ErrorCode catalogue, or pending_confirmation. Add a deterministic grader that judges an observed outcome (post-run data reads, the refusal answer, the pending-action record) against the declared one. It sits next to the merged Run grader (packages/cli/src/lint/scenario-run-grade.ts). The runner Scenario library T3/T4: local Claude Code runner — Build, Run (问数 + operations per persona over the app's MCP), Iterate — writing run records, zero cloud-billed tokens #30292 supplies the observations; this card supplies the judgement.
    2. MCP action-exposure check. The format's run.agentActions declares the actions an agent must be able to call. Check over the built artifact that each declared action exists, is opted into the AI/MCP surface, and is confirm-gated where the card says so. A declared action the artifact does not expose fails; it is never skipped.
    3. Per-persona Run grading. The format computes personaTruths and withheld. The Run grader must grade each persona's answer against that persona's truth, and treat a withheld truth as "must not reveal" (a leak if revealed).
    4. Corpus drift check (carrier note 6651512902): the 3 control cards' expect must stay equal to what packages/cli/src/lint/corpus.ts derives. Pin it where the CLI reads cards.

    Rulings in force (objectstack-ai/cloud#3600): the cost rule 「目前阶段尽量不要用云端的token,api的更贵」 (no model calls; deterministic only); D2 「打分器可以迁移到开源项目」; the declarative floor and execution parameter (comment 6658499884).

  9. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    Claim: PM loop round 2
    Session: session_01Q4EjjPXwX1YfHyKJUh4agu
    Account: hotlong
    Branch: claude/issue-30291-scenario-tag-checks
    Worktree: objectstack-issue-30291
    Domain: domain:server
    Seat: epic:#3600#1
    File surface: packages/cli/src/lint/** (scenario scorer, tag checks, operation grader, Run-grader persona extension, corpus drift pin, and tests), packages/cli/src/commands/lint.ts (the --eval card wiring and its tests), .changeset/30291-scenario-tag-checks.md (stop on breach; explain in the report)
    Container & model: L, mode:subagent, model: TIER_DEFAULT (dispatch-gates --tier: "no path-derived mandate"), effort: xhigh
    Clause-②: no
    Metadata-protocol: none
    Responsibility: n/a — not a defect card
    Thread-read: 6660329073
    Premise: holds
    Serial constraints cleared: none — the 56 open pm:dispatched cards' claimed file surfaces were read and none touches packages/cli/src/lint, packages/cli/src/commands/lint.ts or packages/qa/scenarios; Blocked-by: #30289 and #30290 are both closed (merged 58085ffd, 31d23c3f)

    Clause-②: no is the seat's reading for no new package-entry export. A new os lint flag is CLI surface; the dev measures the build closure and reports. A reading of yes stops the PR for the spec lane's contract review.

  10. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
    os-dev-report

    {
     "issue": 30291,
     "status": "needs_decision",
     "branch": "claude/issue-30291-scenario-tag-checks",
     "pr": "https://github.com/objectstack-ai/objectstack/pull/30594",
     "session": "session_01Q4EjjPXwX1YfHyKJUh4agu",
     "premise_still_valid": true,
     "summary": "Landed as draft PR #30594 (Part of #30291). It covers scope items 1, 3, 4, 5 and 6 as deterministic modules under packages/cli/src/lint: scenario-tags.ts holds the closed 9-row tag table, gradeAgentActions and scoreScenarioArtifact; scenario-operation-grade.ts holds gradeOperation; scenario-run-grade.ts gains gradePersonaRun and truthFigures. The corpus drift pin and an end-to-end score of all 3 control cards are CLI tests that read the card JSON directly. Item 2 is NOT done: os lint --eval cannot read cards in the published CLI. Measured: importing the private @objectstack/scenarios from CLI src builds and passes every dependency gate, but the built module dies with ERR_MODULE_NOT_FOUND on packages/qa/scenarios/src/schema.js (the package ships .ts source with .js specifiers). The package is also private, so publish-smoke-pack never packs it and a published CLI depending on it fails install. Per the dispatch, the schema was not copied; the wiring waits for the decision in open_questions. Zone 2: the MCP opt-in bit is ai.exposed and the confirm gate is ai.requiresConfirmation, both statically readable from the artifact (runtime actionAiExposureError and actionConfirmationRefusal). Handler binding, the automation service and caller permissions are runtime facts left to the runner. The drift-pin assumption held, as an exact expect equality.",
     "tests": "Head f6dfa94e. `pnpm --filter @objectstack/cli exec vitest run --maxWorkers=2 src/lint/` gave Test Files 7 passed (7), Tests 119 passed (119), lock VERDICT command-exit 0. `pnpm --filter @objectstack/cli typecheck` exit 0 (check:test-typecheck: OK, 28 pre-existing ledgered errors, unchanged). Unit layer `vitest run --project unit --maxWorkers=2` at 5760ef6d gave Test Files 392 passed (392), Tests 5489 passed (5489), VERDICT command-exit 0; later commits touched only the cards test (re-run green), the changeset and the input declarations. Integration tier declared to CI (no spawn entry or integration file touched). Ablations via scripts/ablation-replace.mjs: tests import src relatively, so no dist/rebuild leg applies. 18 mutations, every one landed (anchor 1->0, blob moved) and was restored (blob == HEAD, git diff HEAD empty). All red: A01 lookup-resolves 1/30, A02 md-inline 1/30, A03 summary-relates 1/30, A04 options 1/30, A05 fls 2/30, A06 row-scope 1/30, A07 approval-binds 2/30, A09 dashboard 1/30, A10 ai-exposed 2/30, A11 confirm 3/30, A12 unscored-as-pass 1/30, A13 no-write 2/13, A14 unobserved 1/13, A15 audit 2/13, A16 refused-code 1/13, A17 withheld-leak 1/19, A18 scope-leak 2/19. The first A08 (record-triggered-flow-exists -> true) stayed GREEN 30/30 because the check was subsumed by the binding check. Both existence checks were folded into their binding checks (5760ef6d), and the A07/A08 re-run on the binding checks went red (2/30, 1/30). The control-card readings are pinned: crm 10/18, invoice 18/27, project 19/26 (golden corpus has no permission set and no action).",
     "gates": "dispatch-gates --repo objectstack-ai/objectstack --commands on the final diff (11 paths, tree 8947ddd0/f6dfa94e) gave 77 commands, all run, exit codes captured before any pipe. --ran with exit codes: 77 derived, 75 run, 2 NOT-MEASURED, 0 UNRUN. 75 at exit 0 (check:cross-package-test-inputs exit 1 until the declarations were added, then exit 0; ci-filter-parity and turbo-task-graph re-run, exit 0). NOT MEASURED: check:dual-build-cjs-loads exit 3 and check:i18n-coverage exit 3, both PREREQUISITE NOT MET because they read every package's dist and this container has no full build. Left to CI. CI check-runs on PR #30594 at report time are all in_progress (Lint ×9, Type Check ×4, Check Changeset, Governed Surface Queue Guard, PR-body guards); Test Core, Build Core and the UI lanes are not yet listed. NOT MEASURED, not awaited.",
     "dogfood_battery": "n/a — no `!` in the PR title and no (narrowing) arm",
     "mcp_calls": "0",
     "api_writes": "2 — (1) relay pr_create via scripts/pm/fleet-write/dispatch.mjs (repository_dispatch to objectstack-ai/objectstack, run 42071113464, executor POST /repos/objectstack-ai/objectstack/pulls -> #30594, body read back byte-identical, no assignees, no labels); (2) this os-dev-report comment via scripts/pm/post-stamped.mjs (POST /repos/objectstack-ai/objectstack/issues/30291/comments). git push is not counted.",
     "line_budget": "1487 changed lines (+1485 / -2) across 11 files, under the 3000 threshold",
     "files_changed": [
      ".changeset/30291-scenario-tag-checks.md",
      "packages/cli/src/lint/scenario-tags.ts",
      "packages/cli/src/lint/scenario-tags.test.ts",
      "packages/cli/src/lint/scenario-operation-grade.ts",
      "packages/cli/src/lint/scenario-operation-grade.test.ts",
      "packages/cli/src/lint/scenario-run-grade.ts",
      "packages/cli/src/lint/scenario-run-grade.test.ts",
      "packages/cli/src/lint/scenario-cards.test.ts",
      "packages/cli/src/lint/scenario-score.ts",
      "scripts/cross-package-test-inputs.mjs",
      "turbo.json"
     ],
     "clause2_closure": "Clause-②: no, measured. After `pnpm --filter @objectstack/cli build` (tsc, DTS on), walking the relative imports of dist/index.d.ts, dist/console.d.ts and dist/hook-body.d.ts reaches 39 declaration files, 0 under dist/lint/ and 0 under dist/commands/lint. The new modules ship in dist/lint/ but no package entry reaches them; commands/lint.ts is untouched.",
     "deviations": [
      "FILE SURFACE BREACH (claim said stop on breach; I proceeded and declare it here): scripts/cross-package-test-inputs.mjs (+2 globs, +1 heldBy witness, comments) and turbo.json (+2 @objectstack/cli#test inputs). check:cross-package-test-inputs demanded both for the tests the card asks for: the drift pin reads docs/qa/scenarios, and the tag table pin lists docs/qa/platform-checklist/areas. I used the exact glob strings already in the table, so the pinned count in check:ci-filter-parity does not move. The seat may record the surface addition or reject it.",
      "Item 2 (os lint --eval reads cards) and its acceptance bullet are not delivered, so the PR says Part of #30291, not Fixes. The dispatch's stop condition was 'refused by a gate'. No gate refused: undeclared-dep-imports, published-files, workspace-manifest-cycles, dts-closure, changeset-fixed, test-source-alias, override-consistency, lean-entry-closure and vendor-export-contract-resolve all exited 0 with the dependency declared. The refusal is at runtime (ERR_MODULE_NOT_FOUND) and at publish (private package never packed). I stopped on that equivalent evidence and did not copy the schema. Probe reverted: package.json and pnpm-lock.yaml restored from HEAD, probe file deleted, git status clean.",
      "Card slices mirrored as interfaces in the CLI, following the scenario-score.ts precedent with a ⛔ mirror note: ScenarioPersonaSlice, ScenarioAgentAction and ScenarioArtifactCard (scenario-tags.ts), DeclaredOutcome (scenario-operation-grade.ts) and ScenarioTruth (scenario-run-grade.ts). These are type shapes the graders take, not a card parser; no card validation is restated. Flagging this because the dispatch said not to restate the card schema in the CLI.",
      "Four-axis block taken at 71931602 (last SKILL.md commit; unchanged since 04f36b63, per git diff --stat). md5 5434ffb322e7d83da660abec2841468f matches when measured without the trailing newline; with the newline it is abff5f852c8fd09b79ec0623439aa4f0.",
      "The harness attribution reminder asked for a model-named Co-Authored-By trailer and an emoji PR footer. AGENTS.md takes precedence: commits carry the model-free trailer pair, and the PR body uses the session-URL footer form."
     ],
     "open_questions": [
      {
       "question": "How should `os lint --eval` (the published CLI) read scenario cards, given that the card format package @objectstack/scenarios is private, unbuilt (ships .ts with .js specifiers) and never packed? This is the missing half of #30291 (body item 2 and the acceptance bullet 'os lint --eval scores at least one seed card end to end'). Framework: the four-axis block from .claude/skills/pm-dispatch/SKILL.md read at 71931602.",
       "options": [
        "A. Publish @objectstack/scenarios: drop private, add a build step, and have the CLI depend on it; `os lint --eval --scenarios DIR` ships. Business need: measured partly. docs/qa/scenarios/README.md says held-out and customer-derived cards live in a private overlay OUTSIDE this repo, and only a published loader serves that without a copy. Long-term: one format reader, contract-first. AI-error: strict schema stays the only reader. Expansion: a new published package (startup rule leans against it); it also reverses the package's stated 'not published, devx instrument' posture (owner: the scenario lane). Cost: about 1 small PR in packages/qa/scenarios (build and publish config, changeset-fixed membership), then the CLI wiring (about 300 lines).",
        "B. In-repo-only flag: the CLI bundle-requires packages/qa/scenarios/src from the checkout (found by walking up to pnpm-workspace.yaml) and refuses loudly outside a workspace. Business need: serves only in-repo runs, not the private overlay. Long-term: a workaround. AI-error: a published flag that works only in this repo is a declared-but-undelivered capability (Prime Directive 10). Expansion: adds CLI surface with no external user. Cost: low (about 200 lines).",
        "C. Keep the scorer in the CLI and add a subpath export (e.g. @objectstack/cli/scenario-scorer). The card-reading entry lives on the private side (a script or runner in packages/qa/scenarios or the #30292 runner). Business need: in-repo runner only. Long-term: sound layering. AI-error: neutral. Expansion: a new public export (Clause-② yes, spec-lane contract review) for a devx-only consumer. Cost: medium.",
        "D. Move the scenario-* scorer modules from packages/cli/src/lint into @objectstack/scenarios (still in the open repo, so D2 「打分器可以迁移到开源项目」 holds by its words) and drop the os lint --eval card flag from the card. Business need: in-repo only. Long-term: format and scorer co-located, no public surface. Expansion: none (shrinks CLI surface). Cost: medium (git mv plus import rewiring), but it contradicts the dispatch's reading of D2 as placing the scorer in packages/cli and the card body's CLI wiring."
       ],
       "recommendation": "A, if the private overlay outside this repo is a real consumer (the README says it exists). That is the only option that lets the overlay be scored without a second copy of the format, and it keeps one strict reader. If the maintainer rules the overlay is cloud-side and will vendor the format, D is the smallest surface. B is not recommended: it publishes a capability npm users cannot use. The axes conflict on expansion (A adds a published package) versus the measured overlay need, so this is the maintainer's call."
      }
     ],
     "security_findings": "none",
     "out_of_scope_findings": [
      "carrier: seat (no card filed, noted only) · The six mid-tier cards (#30476–#30481) will score only if their tags match the nine rows in CAPABILITY_TAG_CHECKS. Anything else is reported unscored with reason 'no check in the closed tag table'. The devx lane should align tag names or ask for rows. Dedupe words: scenario capability tag names, unscored, closed tag table.",
      "carrier: none · noted only · The golden corpus fixtures (packages/cli/src/lint/corpus.ts) carry no permission set and no action. Each control card's access-security.field-permissions tag and every declared agent action fail against its golden build (crm 10/18, invoice 18/27, project 19/26). That is expected, since the corpus models shape. It is an observation, not one of the three filing classes."
     ]
    }
  11. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    [Decision] 打分器和场景卡读取放在哪个包(dev 的 needs_decision,席位附四维分析)

    一句话问题: 已发布的 os 命令行要读场景卡,但卡片格式包 @objectstack/scenarios 是私有包,而且不构建(交付的是 .ts 源码),已发布的命令行没法依赖它(dev 实测:构建后运行报 ERR_MODULE_NOT_FOUND,打包冒烟测试也不会打包私有包)。

    交付现状: PR #30594(Part of #30291,1487 行)已交付第 1、3、4、5、6 项:能力标签检查(9 行封闭表)、写入操作判分、agent 动作暴露检查、按角色判分、语料漂移钉子。未交付的是第 2 项:os lint --eval 读场景卡。

    Governing text:

    • cloud#3600 D2「打分器可以迁移到开源项目」(只说迁到开源,没说放进命令行;"放进 os 命令行"是席位当初的建议);
    • 本卡正文第 2 项;
    • NORTH-STAR「创业阶段不扩散」和 AGENTS.md 的 declared ≠ enforced(不发布一个装上就用不了的能力)。

    Prior rulings read:D2 / scorer / os lint --eval → cloud#3600 D2(正文)、6658499884;thread: none

    席位补的一条实测(dev 漏了): cloud 是通过 link: 指向本机的框架固定版本目录来使用框架包的(例如 "@objectstack/cli": "link:../../../objectstack/packages/cli"),不经过 npm。所以 cloud 的私有补充层要读场景卡,并不需要发布任何东西,直接 link 到 packages/qa/scenarios 就行。这推翻了 dev 推荐 A 的主要理由("只有发布的读取器才能服务私有补充层")。

    选项 做什么 后果 工作量
    A 把 @objectstack/scenarios 改成公开发布包,命令行依赖它,os lint --eval --scenarios 对外提供 多一个对外发布的包和对外承诺;目前没有外部用户需要 1 个小 PR 加约 300 行接线;无先例
    B 命令行只在本仓源码目录里找场景卡格式,在仓外调用就拒绝 发布了一个 npm 用户用不了的能力(declared ≠ enforced) 约 200 行
    C 打分器留在命令行,加一个子路径导出(已立卡 #30522,条款② 为 yes);读卡放在私有侧(执行器 #30292) 已发布的命令行多一个对外导出,消费方只有内部 中等;与 #30522 合并做
    D 把 scenario-* 打分器从 packages/cli/src/lint 挪进 @objectstack/scenarios(仍在开源仓,D2 照样成立),本卡第 2 项"命令行读卡"取消;#30522 随之不需要,关为 not planned;cloud#3604 改为 link 引入这个包 对外发布面零增加,反而缩小:格式、读取器、打分器放在同一个包,只有一份 中等:挪 5 个模块和测试、改引用;#30594 改成在新位置落地

    业务含义: A 是先开一家对外营业的门店再看有没有顾客;D 是先把厨房和菜单放在同一个后厨,真有外卖需求再开门店(届时把这一个包公开发布即可,就是 A)。

    os-decision-facets

    • ① 项目长远合理性:D 收窄,格式和打分器同处一个包,没有镜像接口(dev 这轮为避免依赖私有包,不得不在命令行里镜像了 5 个卡片类型,D 下这些重复都会消失);A 扩大,新增一个发布包。
    • ② 实际业务拉动:实测的消费方只有本仓执行器和 cloud 补充层,两者都靠 link 使用,不需要发布;外部 npm 用户零拉动。
    • ③ 防 AI 犯错:D 只有一个严格的读取器和一套打分器;C 和 A 需要跨包对齐;B 声明了却兑现不了。
    • ④ 创业阶段不扩散:D 不新增公开接口、不新增发布包;A、C 都新增一个公开承诺。

    推荐 D。 两年后的样子:场景库(格式 + 读取器 + 打分器)是一个独立的包,开源;有外部用户时整体发布,对标 Salesforce 的 package 校验工具链放在独立工具包里,而不是塞进主命令行。

    置信缺口: cloud 用 link 引用一个"不构建、交付 .ts 源码"的私有包,在 cloud 的测试运行器里能否直接解析,我没有实测(cloud 现有的 link 都指向有构建产物的包)。D 落地前要让 #3604 的车道先做一次可解析性实测;如果解析不了,就给场景包补一个最小的构建步骤,仍不发布。

    自检:只看①选 D;②③④ 是否翻转:否。

    裁后执行(若选 D):

    1. feat(scenarios): the scenario scorer moves into @objectstack/scenarios and judges capability tags, agent-action exposure, write outcomes and per-persona answers #30594 补丁轮:把 scenario-score/graph/run-grade/tags/operation-grade 及测试挪进 packages/qa/scenarios;在私有包里提供"读卡并打分"的入口,供执行器调用;去掉命令行里的镜像类型;本卡改为 Fixes #30291,第 2 项在卡上记为按裁决取消。
    2. Export the scenario scorer (scoreGraph / scoreGraphShape / deriveShape / artifactToEvalGraph / gradeRun) from a published entry so cloud can drop its drifting copy #30522 关闭,原因 not planned(被本裁决取代)。
    3. cloud#3604 的阻塞改为"本 PR 合并 + cloud 的框架版本更新",附上 link 可解析性实测这一前提。
    4. Scenario library T3/T4: local Claude Code runner — Build, Run (问数 + operations per persona over the app's MCP), Iterate — writing run records, zero cloud-billed tokens #30292 执行器直接依赖 @objectstack/scenarios。

    Generated by Claude Code

  12. added
    needs-user-decisionBlocked on a maintainer decision — do not dispatch; decision cards only, never on a PR
    and removed
    pm:dispatchedDispatched to a dev agent by /pm-dispatch
    on Nov 5, 2026
  13. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    Ruled: D

    Maintainer, verbatim (Claude Code session, 2026-11-05): 「同意 D」, on decision comment 6664312112.

    Execution (all four steps in this round):

    1. This card returns to flight (needs-user-decision → pm:dispatched, same claim, same branch). PR feat(scenarios): the scenario scorer moves into @objectstack/scenarios and judges capability tags, agent-action exposure, write outcomes and per-persona answers #30594 gets a patch round:
      • move scenario-score, scenario-graph, scenario-run-grade, scenario-tags, scenario-operation-grade and their tests from packages/cli/src/lint/ into packages/qa/scenarios;
      • add a private "read a card and score it" entry for the runner;
      • drop the CLI's mirrored card types;
      • body item 2 (os lint --eval reads cards) is cancelled by this ruling, and the PR becomes Fixes #30291.
      • File surface amended: add packages/qa/scenarios/**, plus removal of the moved files from packages/cli/src/lint/ (including those merged in PR feat(cli): port the deterministic scenario scorer — shape score, artifact graph reader and Run grader — with a discrimination self-test #30332) and any CLI test that imported them. The round-2 breach (scripts/cross-package-test-inputs.mjs, turbo.json, gate-forced) is recorded here as accepted.
    2. objectstack#30522 (the scorer export) is unneeded under D. It is claimed in flight by domain:devx#1, so the seat does not touch it. The ruling is sent to that seat's mailbox (Notify: ruling) and noted on the card; that seat releases or closes it.
    3. objectstack-ai/cloud#3604 now waits on this card's PR plus the pin. A precondition is added: measure that cloud resolves @objectstack/scenarios (.ts source, no build) through its link: path.
    4. Scenario library T3/T4: local Claude Code runner — Build, Run (问数 + operations per persona over the app's MCP), Iterate — writing run records, zero cloud-billed tokens #30292 (runner) depends on @objectstack/scenarios directly.
  14. added
    pm:dispatchedDispatched to a dev agent by /pm-dispatch
    and removed
    needs-user-decisionBlocked on a maintainer decision — do not dispatch; decision cards only, never on a PR
    on Nov 5, 2026
  15. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu
    os-dev-report

    Patch round (ruling D). Report:

    {
     "issue": 30291,
     "status": "done",
     "branch": "claude/issue-30291-scenario-tag-checks",
     "pr": "https://github.com/objectstack-ai/objectstack/pull/30594",
     "session": "session_01Q4EjjPXwX1YfHyKJUh4agu",
     "premise_still_valid": true,
     "summary": "Patch round under ruling D (comment 6670490510), head 4dd3edc5 after one merge of origin/main (b77c160a). The 5 scorer modules and 6 test files moved by git mv from packages/cli/src/lint into packages/qa/scenarios/src/score and test/score, so renames are detected and history follows. The CLI keeps no scenario code and no mirrored card types. The graders now take the card schema's own types: ScenarioExpect, z.infer of OutcomeSchema and PostStateSchema, a Pick of ScenarioCard, and the loader's Truth. New private subpath @objectstack/scenarios/score holds scoreCardArtifact, gradeCardOperation and gradeCardAnswer, all taking a LoadedScenario, for the runner #30292. resolveStackCollection comes from its owner @objectstack/core. Deps core, objectql and plugin-sharing are declared and aliased to source. The closed 9-row tag table is listed in packages/qa/scenarios/README.md. The PR becomes Fixes #30291; the replacement body is attached below for the seat to write. One enabling CLI edit: MetadataEvalCase moved into corpus.ts and is re-exported by metadata-eval.ts, so the scenarios type-check program does not pull in the lint command when its tests import the corpus.",
     "tests": "Head 4dd3edc5. `pnpm --filter @objectstack/scenarios exec vitest run --maxWorkers=2` gave Test Files 11 passed (11), Tests 163 passed (163); no file above 0.2 s of test wall. `pnpm --filter @objectstack/scenarios typecheck` exit 0. The core pin second-object-read-exposure.pin.test.ts gave 4/4. `pnpm --filter @objectstack/cli typecheck`, build and `vitest run test/metadata-eval.test.ts src/lint/` gave 43/43 under lock VERDICT command-exit 0. The CLI unit layer was not re-run in full this round; it was green (392/392 files) at 5760ef6d, and this round's CLI change is type-only (corpus.ts, metadata-eval.ts) plus file removals. CI Test Core owns the full run. One ablation per moved module at the new home, via scripts/ablation-replace.mjs (tests import src relatively, so no dist leg). Every mutation landed and was restored (blob == HEAD, git diff HEAD empty), and every one went red: scenario-score measure 2/16, scenario-graph nav-resolves 2/5, scenario-run-grade withheld 1/19, scenario-tags md-inline 1/30, scenario-operation-grade no-write 2/13, score/index wrong-outcome 1/7.",
     "gates": "dispatch-gates --repo objectstack-ai/objectstack --commands on the patch diff: 91 commands. --ran with exit codes: 91 derived, 89 run, 2 NOT-MEASURED, 0 UNRUN. RED: `node scripts/check-empty-changeset.mjs --base origin/main` exit 1, the deliberate-correction refusal on the deleted unreleased .changeset/30290-scenario-scorer-port.md (see open_questions; per the gate's own text it stays red until a person confirms). NOT MEASURED: check:dual-build-cjs-loads and check:i18n-coverage, both exit 3 PREREQUISITE NOT MET (no full build here); CI owns them. Every other family exit 0, including cross-package-test-inputs, ci-filter-parity, turbo-task-graph, undeclared-dep-imports, test-source-alias and workspace-manifest-cycles. CI check-runs for 4dd3edc5 not read, not awaited.",
     "dogfood_battery": "n/a — no `!` in the PR title and no (narrowing) arm",
     "mcp_calls": "0",
     "api_writes": "1 this round: this os-dev-report addendum via scripts/pm/post-stamped.mjs (relay; POST /repos/objectstack-ai/objectstack/issues/30291/comments). PR body not written (the seat writes it from the attachment). git push not counted. Round-2 writes (pr_create, report comment 6664223112) were reported in round 2.",
     "line_budget": "GitHub-method (rename-aware, git diff -M --numstat vs merge base b77c160a): +1741 / -65 = 1806, under 3000. dispatch-gates reads 4324 (+3000 / -1324) because it counts with --no-renames, so every moved file counts as a full delete plus add. GitHub counts with rename detection, measured on two merged controls: #30375 is no-renames 6567 / renames 1561 / GitHub 1561; #30550 is 3228 / 2482 / GitHub 2482. I pushed on the GitHub-method reading.",
     "files_changed": [
      ".changeset/30290-scenario-scorer-port.md (deleted)",
      ".changeset/30291-scenario-tag-checks.md",
      "packages/cli/src/lint/corpus.ts",
      "packages/cli/src/lint/metadata-eval.ts",
      "packages/core/src/security/second-object-read-exposure.pin.test.ts",
      "packages/qa/scenarios/README.md",
      "packages/qa/scenarios/package.json",
      "packages/qa/scenarios/vitest.config.ts",
      "packages/qa/scenarios/src/score/{index,scenario-graph,scenario-operation-grade,scenario-run-grade,scenario-score,scenario-tags}.ts",
      "packages/qa/scenarios/test/score/{entry,scenario-cards,scenario-graph,scenario-operation-grade,scenario-run-grade,scenario-score,scenario-tags}.test.ts",
      "packages/cli/src/lint/scenario-*.ts and scenario-*.test.ts (renamed away)",
      "pnpm-lock.yaml",
      "scripts/cross-package-test-inputs.mjs",
      "turbo.json"
     ],
     "clause2_closure": "Clause-②: no, re-measured after a fresh CLI build. Walking the relative imports of dist/index.d.ts, dist/console.d.ts and dist/hook-body.d.ts reaches 39 files: 0 scenario files and no bare @objectstack/scenarios specifier. dist/lint holds only corpus, hook-body-lowering, metadata-eval and score. @objectstack/scenarios stays private; its new ./score subpath is a private-package entry.",
     "deviations": [
      "Outside the amended surface: packages/cli/src/lint/corpus.ts and metadata-eval.ts. MetadataEvalCase moved into corpus.ts and is re-exported under the same name. Reason: the moved tests import the CLI corpus, and corpus.ts's type import from metadata-eval.ts would pull score.ts and then commands/lint.ts into the scenarios type-check program. No behaviour change.",
      "Deleted the unreleased .changeset/30290-scenario-scorer-port.md, which belongs to merged PR #30332. It announces 'feat(cli): a deterministic scenario scorer under src/lint', which this PR makes false; restoring it would publish that false CLI changelog line. check-empty-changeset reds on it by design (deliberate-correction class) until a person confirms; see open_questions.",
      "Also outside the named surface but forced by gates: packages/qa/scenarios/vitest.config.ts (check:test-source-alias demanded source aliases for core, objectql and plugin-sharing) and pnpm-lock.yaml (the three new workspace deps).",
      "Line budget: the repo tool says OVER (4324), the GitHub method says 1806. I pushed on the GitHub method, measured against two controls (see line_budget). If the seat reads the tool's number as binding, this is the item to reverse.",
      "The replacement PR body is attached with its details/summary folds turned into ### subsections, so the comment sanitizer cannot eat tag-shaped fragments. The seat may fold them back when writing the body."
     ],
     "open_questions": [
      {
       "question": "Confirm the deletion of the pending release note .changeset/30290-scenario-scorer-port.md? It describes CLI modules that this PR moves into the private @objectstack/scenarios, and check-empty-changeset stays red on this deliberate-correction class until a person confirms on the PR.",
       "options": [
        "A. Confirm the deletion. The new .changeset/30291-scenario-tag-checks.md states the move.",
        "B. Restore it from the merge base. The gate goes green, but the next release publishes 'feat(cli): a deterministic scenario scorer under src/lint', which is false."
       ],
       "recommendation": "A. The gate's own text says restoring republishes a now-false sentence and that no diff shape makes that safe."
      }
     ],
     "security_findings": "none",
     "out_of_scope_findings": [
      "class: a · reach: named producer scripts/pm/dispatch-gates.mjs changedLinesFromGit (and check-governed-merges.mjs --branch mode), both reading `git diff --numstat --no-renames` · evidence: on a rename-heavy diff the size reading disagrees with GitHub's PR additions+deletions, which check-governed-merges.mjs's header names as the authority. #30375 reads 6567 locally vs GitHub 1561; #30550 3228 vs 2482; this PR 4324 vs a rename-aware 1806. The tool tells a dev to stop at the 3000-line human-merge threshold when GitHub would not cross it · dedupe words: dispatch-gates changed lines no-renames, human-merge threshold rename, numstat --no-renames size",
      "carrier: seat (noted, not filed) · The merged travel_expense.json card declares 12 tags. 5 score; 7 are unscored against the closed table: records-forms.formula-field, records-forms.validation-rule, approvals.multi-step-approval, access-security.row-level-security, dashboards.report, ai.ask-data, ai.agent-actions. The table is now in packages/qa/scenarios/README.md for the remaining five mid-tier authors to align to, or to extend with a row plus a discrimination test."
     ]
    }

    Replacement PR body for #30594 (for the seat to write)

    Fixes #30291
    Clause-②: no
    Metadata-protocol: none
    
    ## 维护者速读
    
    - 改了什么:确定性场景评分器整体从 `packages/cli` 迁入私有包 `@objectstack/scenarios`(裁决 D)。新增封闭的能力标签检查表、Agent 动作暴露检查、写操作结果判分和按角色判分,并提供给运行器用的入口 `@objectstack/scenarios/score`。
    - 为什么改:维护者裁定方案 D(「同意 D」),评分器和卡片格式放在一起。已发布的 CLI 不再携带任何场景代码,也不再需要依赖私有包。
    - 风险与回滚:CLI 只把 `MetadataEvalCase` 类型挪到 `corpus.ts` 并原名重导出,行为不变。评分器此前没有从 CLI 发布过。回滚即撤销本 PR。
    - 先看:`packages/qa/scenarios/src/score/scenario-tags.ts`(标签表与动作暴露判定)、`packages/qa/scenarios/src/score/index.ts`(运行器入口)、`.changeset/30290-scenario-scorer-port.md` 的删除(见下方"需要你做的")。
    - 两分钟验证:`pnpm --filter @objectstack/scenarios test`,预期 11 个文件、163 个用例全部通过。
    - 需要你做的:确认删除尚未发布的 `.changeset/30290-scenario-scorer-port.md`。它写的是"CLI 新增评分器",迁移后这句话已不成立。`check-empty-changeset` 对这类有意更正会一直保持红色,直到有人确认。
    
    ## What changes
    
    Per the maintainer's ruling D on #30291 (「同意 D」), the deterministic scenario scorer lives in the private `@objectstack/scenarios` package (`packages/qa/scenarios`), beside the card format it judges. No model call anywhere (cost ruling on objectstack-ai/cloud#3600: 「目前阶段尽量不要用云端的token,api的更贵」).
    
    - **Moved with `git mv`, so history follows:** `scenario-score`, `scenario-graph` and `scenario-run-grade` (merged in #30332), `scenario-tags` and `scenario-operation-grade`, and their tests. They go from `packages/cli/src/lint/` to `packages/qa/scenarios/src/score/` and `test/score/`. `@objectstack/cli` ends with no scenario code and no mirrored card types.
    - **The graders take the card schema's own types:** `ScenarioExpect`, `z.infer` of `OutcomeSchema` and `PostStateSchema`, a `Pick` of `ScenarioCard`, and the loader's `Truth`. The `expect`, outcome, persona, agent-action and truth mirrors are gone.
    - **`@objectstack/scenarios/score`** (private subpath) is the runner's entry (#30292). It has three functions, and each takes a card the loader returned:
      - `scoreCardArtifact(card, artifact)`;
      - `gradeCardOperation(card, operationId, observation)`;
      - `gradeCardAnswer(card, { question, persona?, turn })`. The persona's hidden objects and fields are read off the card.
    - **Capability-tag checks** (`scenario-tags.ts`): a closed table of nine rows, listed in the package README for the mid-tier card authors.
      - A tag with no row is `unscored`, never a pass. So is a row whose card gives it nothing to judge. Both are counted nowhere.
      - A row that runs zero checks fails.
      - Rows read the owning predicates (`resolveFlowTriggerKind`, `APPROVAL_NODE_TYPE`, `SINGLE_OPTION_TYPES`, `summaryFunctionReadsField`, `effectiveSharingModel`, `standaloneActionOwnerKey`). Collections are read through `resolveStackCollection` from its owner, `@objectstack/core`.
    - **Agent-action exposure:** each declared action must exist on its object, carry `ai.exposed: true` (the MCP doors' opt-in) and, where the card says `confirm`, carry `ai.requiresConfirmation: true` (the flag the confirmation gate enforces, not the delete-mode listing heuristic). A missing action fails all of its checks.
    - **Write-operation grader:**
      - applied: every post-state is read back, one row per `where`, and the audit actor matches;
      - refused: the exact `ErrorCode`, with zero writes;
      - pending_confirmation: zero writes.
    
      An unobserved fact fails.
    - **Per-persona Run grading:**
      - a `withheld` truth is must-not-reveal: a decline is correct, and a stated all-rows figure is a leak;
      - an all-rows figure outside a scoped persona's rows is a leak when the reply states it.
    - **Dependencies** of the private package: `@objectstack/core`, `@objectstack/objectql`, `@objectstack/plugin-sharing` (workspace). Its vitest config aliases each to source (`check:test-source-alias`).
    - **Gates re-homed:**
      - the cross-package inputs move from `@objectstack/cli#test` to the scenarios entry and task, which also declare `packages/cli/src/lint/corpus.ts` (the golden builds and the drift pin read it);
      - the `second-object-read-exposure` pin row for `scenario-graph.ts` follows the file (same `not-a-served-read` class).
    - **CLI:** `MetadataEvalCase` is declared in `corpus.ts`, beside the corpus it types, and `metadata-eval.ts` re-exports it under the same name. That keeps the scenarios type-check program from pulling in the whole lint command.
    - **Changesets:** one changeset (`@objectstack/scenarios` and `@objectstack/cli`, patch). The unreleased `.changeset/30290-scenario-scorer-port.md` is **deleted on purpose**: it announces a scorer in the CLI that this PR moves out, so it is a deliberate correction of a pending release note. `check-empty-changeset` stays red on it until a person confirms.
    - Body item 2 (`os lint --eval` reads cards) is cancelled by the ruling.
    
    ## Tests
    
    - `pnpm --filter @objectstack/scenarios test`: 11 files, 163 tests. Moved suites unchanged in what they assert. Added `test/score/entry.test.ts` (the runner entry over the real `crm_account_with_contacts` card).
    - `scenario-cards.test.ts` loads the control cards through the loader:
      - **corpus drift pin.** Each control card's `expect` must equal what `corpus.ts` derives, with a floor of exactly 3 controls;
      - **end-to-end readings:** crm 10/18, invoice 18/27, project 19/26;
      - crm completed with what it asks scores 1, and each removal scores lower.
    - `pnpm --filter @objectstack/scenarios typecheck`: exit 0.
    - `pnpm --filter @objectstack/core exec vitest run src/security/second-object-read-exposure.pin.test.ts`: 4/4.
    - `pnpm --filter @objectstack/cli typecheck`, build, and `vitest run test/metadata-eval.test.ts src/lint/`: 43/43.
    
    ## Acceptance notes
    
    - **Tag names.** `travel_expense.json` (merged meanwhile) declares 12 tags. Five score: `master-detail-inline-edit`, `roll-up-summary`, `select-options`, `automation.record-triggered-flow` and `access-security.field-permissions`. The rest are `unscored`: `records-forms.formula-field`, `records-forms.validation-rule`, `approvals.multi-step-approval`, `access-security.row-level-security`, `dashboards.report`, `ai.ask-data` and `ai.agent-actions`. The README table is what the remaining five card authors align to, or extend.
    - **Size.** `dispatch-gates` reads 4324 changed lines: it counts with `--no-renames`, so every moved file counts twice. The rename-aware count is 1806, and that is how GitHub counts. Two controls: #30375 is 6567 no-renames vs 1561 rename-aware vs GitHub 1561; #30550 is 3228 vs 2482 vs GitHub 2482.
    - The round-2 gate-forced additions to `scripts/cross-package-test-inputs.mjs` and `turbo.json` are recorded as accepted on #30291. They now point at the scenarios package.
    
    ## Verification
    
    ### One ablation per moved module, at the new home (scripts/ablation-replace.mjs; tests import src relatively, so no dist leg)
    
    Every mutation landed (anchor 1 → 0) and was restored (blob == HEAD, `git diff HEAD` empty):
    
    | module | mutation | result |
    |:---|:---|:---|
    | scenario-score | widget measure check → pass | 2 failed / 16 |
    | scenario-graph | nav entry resolves → true | 2 failed / 5 |
    | scenario-run-grade | withheld leak scan off | 1 failed / 19 |
    | scenario-tags | master-detail inlineEdit → true | 1 failed / 30 |
    | scenario-operation-grade | no-write → true | 2 failed / 13 |
    | score/index | operation judged against the wrong outcome | 1 failed / 7 |
    
    The full per-check battery of the previous round (18 mutations) ran against the same assertions before the move.
    
    
    ### Gates
    
    - `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands` on the patch-round diff: 91 commands, all run, exit codes captured before any pipe.
    - `check-empty-changeset --base origin/main`: exit 1, the deliberate-correction refusal on `30290-scenario-scorer-port.md` (above).
    - `check:dual-build-cjs-loads` and `check:i18n-coverage`: exit 3 `PREREQUISITE NOT MET`. They read every package's built output; NOT MEASURED, CI owns them.
    - Every other family: exit 0, including `check:cross-package-test-inputs`, `check:ci-filter-parity`, `check:turbo-task-graph`, `check:undeclared-dep-imports`, `check:test-source-alias` and `check:workspace-manifest-cycles`.
    - Clause-② closure: walking the relative imports of the CLI's `dist/index.d.ts`, `dist/console.d.ts` and `dist/hook-body.d.ts` reaches 39 files, none a scenario file, and no bare `@objectstack/scenarios` specifier. `dist/lint/` holds only `corpus`, `hook-body-lowering`, `metadata-eval` and `score`.
    - Dogfood: n/a. No `!` in the title and no `(narrowing)` arm.
    
    
    ---
    _Generated by [Claude Code](https://claude.ai/code/session_01Q4EjjPXwX1YfHyKJUh4agu)_
  16. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    ACCEPT → PR #30594 at head 4dd3edc5. The contract review passed, and the DELIBERATE CORRECTION of the pending .changeset/30290-scenario-scorer-port.md is confirmed on the PR. Landing through the merge queue with the designed Check Changeset red (all three conditions recorded).

  17. objectstack-fleet commented on Nov 5, 2026

    @objectstack-fleet
    ContributorAuthor

    epic:#3600#1 · hotlong · session_01Q4EjjPXwX1YfHyKJUh4agu

    Landed: PR #30594 merged as 6ef7f583, closing this card (Fixes #30291; body item 2 cancelled by ruling D). pm:dispatched removed with the close. Unblocked by this landing: the runner #30292 (both its Blocked-by: lines now closed), dispatched by the epic PM as a cloud card. objectstack-ai/cloud#3604 now waits only on the .objectstack-sha pin carrying 6ef7f583.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:devpathThe road — create, dev, verify, publish/install, connect an agent, iteratedomain:serverDomain lane — seat card indexed by label:pm:seatpm:blockingDerived cache from the Blocked-by reverse index: open card with open dependents — never hand-setpm:epicReserved by a dedicated epic PM, parent or sub-issue; other PMs never take it; never with pm:queuepriority:p1High: required for production / M2

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions