Skip to content

skills(automation): integrations are declared, not coded — rule section, per-archetype recipes, three evals - #27322

Merged
os-zhuang merged 2 commits into
mainfrom
claude/issue-27223-integrations-declared
Oct 28, 2026
Merged

os-zhuang merged 2 commits into
mainfrom
claude/issue-27223-integrations-declared

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Fixes #27223

Clause-②: no
Metadata-protocol: none

维护者速读

  • 改了什么:objectstack-automation 技能包新增规则节「Integrations are declared, not coded」(厂商系统走应用内声明的 connectors: 条目 + connector_action,回调走 api 流;五种情形才写平台代码;加载器暂不支持的文档是「待平台卡」的缺口,不是写插件的理由),connector_action 节点行不再写 "pre-built";新增 references/integration-recipes.md(六种原型各一份配方 + 作者/管理员凭据交接 + 两条「尚未支持」)和 evals/integrations-declared.json(DocuSign / Stripe 读 / 无规格厂商三例,DocuSign 例对任何「写平台插件」的答案判失败)。
  • 为什么改:HotCLM 的开发 agent 读了技能后结论是平台要造 DocuSign 插件(维护者原话 「hotclm 的开发agent 就搞错了,说明 skills还是不对,要立卡完善」);技能里确有两句把它推向这个方向。
  • 风险与回滚:纯技能文本与 eval,无运行时改动;token 棘轮:SKILL.md 净 −1 token,两份新文件按落地值钉死(整包 +3370 token);回滚即 revert 本 PR。
  • 先看:① skills/objectstack-automation/SKILL.md 新节(约第 402 行起)——五种「写平台代码」的情形与缺口命名的措辞;② references/integration-recipes.md 「Holds on every recipe」第二条(connectorConfig.input 按原样发送,逐记录 payload 今天不可声明)——这是实测到的运行时事实,也是下方 Acceptance notes 第 1 条发现;③ evals/integrations-declared.json 第 1 例的 must_not_contain。
  • 两分钟验证:node scripts/check-skills-token-ratchet.mjs && node scripts/pm/check-skill-id-lint.mjs——预期两条均 ✓,automation 五个计价文件 headroom 全为 0。
  • 需要你做的:Tier H 技能面,由你手动落地(merge);卡号为何未写进技能正文见 What changes 末段。

What changes

  • skills/objectstack-automation/SKILL.md — the connector_action node row no longer says "pre-built integration connector"; the "reserve custom code for edge-case integrations" sentence is gone; a new section Integrations are declared, not coded states the rule, the five cases that justify platform code, and that a document the loader cannot take yet (YAML, Swagger 2.0, $ref, OAuth, request signing) is a gap to name against its platform card, never a reason to write code. The http row's auth-routing tail moved into the recipes. Paid for in the same file: net −1 token under the ratchet, headroom 0.
  • skills/objectstack-automation/references/integration-recipes.md (new, 150 lines) — one recipe per archetype: OpenAPI with a static key (Stripe balance, nightly), remote MCP with a static header (GitHub), plain REST (provider: 'rest', actionId: 'request'), the vendor callback (api flow with secretRef), a hand-written OpenAPI 3.x document for a spec-less vendor, and the two not yet entries (OAuth-protected, signature-authenticated). Each states what the app author declares and what the customer's administrator supplies. It also carries three limits measured on this base: connectorConfig.input is handed to the provider as written (no CEL envelope or {{ }} hole is evaluated — service-automation/src/builtin/connector-nodes.ts:108, and value-slot-map.ts lists no connector slot), the openapi provider has no include key (connector-openapi/src/openapi-provider.ts:38-52 — the trimmed document is the allowlist), and the inbound api door verifies only x-objectstack-signature in the timestamped form (trigger-api/src/api-trigger.ts:163-178, "has no unsigned branch"), so a vendor signing its own header cannot post today.
  • skills/objectstack-automation/evals/integrations-declared.json (new) — the DocuSign case (expected: a declarative openapi instance over a trimmed, packaged 3.x document converted offline from Swagger 2.0, an api callback flow, three named gaps, marked blocked on OAuth; must_not_contain refuses registerConnector(, connector-esign, provider: 'docusign', implements Plugin, clientSecret, token:), the Stripe-read case (scheduled flow writing crm_account fields) and the vendor-without-spec case.
  • scripts/check-skills-token-ratchet.mjs — the SKILL.md row re-locked at its measurement (6466 → 6465); two rows pin the new files at their landed counts (1193, 2177).
  • .changeset/27223-integrations-declared.md — @objectstack/skills patch (the catalog ships in that package's dist via sync-catalog.mjs).

Card numbers are not in the skill text. The card asks the recipes to name the lifting cards. pnpm check:pm-skill-id-lint forbids #NNN in every .md under skills/ by maintainer ruling (2026-08-12, widened to the published catalog on 2026-10-12) — measured on this tree: a probe line Waits on #26709 for include. under references/ reds the gate (exit 1, naming the file and line). The recipes therefore name each gap and say it is blocked on "its platform card"; the numbers live here: include on the openapi provider and offline normalization → #26709; oauth2 in the declarative auth union → #26702; request signing → #26710; the administrator-filled credential object → #27237 (in flight; the recipes teach today's env-var credentialRef only).

Bundle growth, with its authorization. The published bundle grows by 3,370 tokens (two new files pinned at landed counts; SKILL.md −1). The ratchet lands growth only against a maintainer ruling quoted in the PR body; this one is the card's own directive, verbatim and untranslated: 「所以应该先处理 A/B 吧,并且要有 skills或者文档指导元数据开发者。前面agent还在建议我开发 DocuSign」 and 「hotclm 的开发agent 就搞错了,说明 skills还是不对,要立卡完善」(2026-10-28). The PM-set budget — SKILL.md net tokens ≤ 0, recipes ≤ 150 lines, one eval file — is met. The claim's "one entry in references/_index.md" does not apply: that file is generator-owned (gen:skill-refs writes schema pointers only); check:skill-refs passes with the hand-written reference beside it, and SKILL.md links the recipes directly.

Tests

No package source changed, so no pnpm test / pnpm typecheck is owed; the edited gate script's own suite is its --self-test (79 cases pass, below).

Skill evals — what exists and what was run. The root repo has no eval runner: no pnpm script consumes skills/*/evals/*.json, and the only eval gate in the tree, ui/scripts/check-skill-eval-tokens.mjs, scans ui/skills alone. The card's "pnpm skill evals green" therefore cannot be run as spelled. What was run instead, from a scratch replay (not committed) of that gate's whole-token rule — every must_contain token must occur as a whole token in the bundle's .md corpus — over skills/objectstack-automation/:

Oracle, base bundle (a0aa8a1): the three new evals FAIL — exit 1
corpus: 5 .md file(s): SKILL.md, evals/approvals/test-revise-loop.md, references/_index.md, references/examples-flows.md, references/state-machines-and-approvals.md
FAIL eval 1: 8 must_contain token(s), 5 untaught -> "defineConnector", "provider: 'openapi'", "type: 'api'", "Swagger 2.0", "blocked"
FAIL eval 2: 7 must_contain token(s), 1 untaught -> "provider: 'openapi'"
FAIL eval 3: 5 must_contain token(s), 3 untaught -> "provider: 'openapi'", "operationId", "type: 'basic'"
ORACLE: 9 must_contain token(s) untaught
Oracle, this branch (2993d73): the three new evals PASS — exit 0
corpus: 6 .md file(s): SKILL.md, evals/approvals/test-revise-loop.md, references/_index.md, references/examples-flows.md, references/integration-recipes.md, references/state-machines-and-approvals.md
PASS eval 1: 8 must_contain token(s), 0 untaught
PASS eval 2: 7 must_contain token(s), 0 untaught
PASS eval 3: 5 must_contain token(s), 0 untaught
ORACLE: every must_contain token is taught by the bundle

NOT MEASURED: a live LLM answering arm per eval (base text vs new text) — no runner exists and a self-graded run by the authoring seat is not evidence; the DocuSign case's refusal of a plugin answer is enforced by its must_not_contain list, not by a measured model run.

Acceptance notes

  1. Finding, class c (metadata-authoring trap), for the seat to file. connectorConfig.input on a connector_action node is handed to the provider handler as written (packages/services/service-automation/src/builtin/connector-nodes.ts:108: handler((cfg.input ?? {}) …)); builtin/value-slot-map.ts resolves fields, subflow/map input, script inputs, screen defaults, notify templateData and http body, never connector input. The authoring door accepts a CEL envelope there: FlowSchema.safeParse of a flow whose connectorConfig.input is { path: { customer: { dialect: 'cel', source: 'record.stripe_customer_id' } } } answers success = true and keeps the envelope verbatim (measured with tsx against packages/spec/src/automation/flow.zod.ts on this tree). The openapi handler then interpolates the path from String(value), so the vendor receives an object stringified. Reach: os validate runs FlowSchema; producer: any app author's stored flow definition. Dedupe words: connector_action input value slot CEL envelope verbatim. The recipes state the limit; they do not work around it.
  2. The card's "pnpm skill evals" does not exist for the root catalog — the eval fixtures under skills/*/evals/ have no mechanical consumer in this repository; ui/ has one for its own bundle. Observation for the PM (the acceptance wording), not a card from this seat.
  3. Pre-existing, observation only: under the same whole-token rule, two existing evals in evals/flows-triggers-approvals.json carry untaught tokens on base as well as on this branch — eval 7 (excludeSubmitter: true) and eval 8 (type: 'screen', type: 'file', signed_copy). No root gate asserts the rule; carrier: none.
  4. #25747 (the DocuSign instance the card cites as open) is closed not_planned as of 2026-10-28 — no effect on the premise, which was re-verified on origin/main a0aa8a1: pre-built integration connector at SKILL.md:89; defineConnector|provider: 'openapi' → 0 files under skills/*/references and skills/*/evals; Declaring a provider-bound instance at content/docs/automation/connectors.mdx:40.
  5. references/examples-flows.md:49-55 (the plugin-registered slack connector in a connector_action) is outside the declared surface and was not edited; it stays a legitimate shape, and the rewritten connector_action row says so ("or a plugin's").
  6. The runAs: 'system' paragraph in SKILL.md says a system sweep must pin its organization but names no column, and the catalog's own scheduled example carries no predicate; the Stripe recipe mirrors the taught shape and adds a one-line reminder. Read-only inference; noted, not filed.
  7. Five derived families exit 3 locally (PREREQUISITE NOT MET: they read built dist/, which this diff never builds) — check:doc-formula-expressions, check:dts-closure, check:dual-build-cjs-loads, check:lean-entry-closure, check:sourcemap-no-sources-content. They are pulled in by the scripts/ touch, read none of the changed paths, and run in CI after the build; recorded as NOT MEASURED locally.

Verification

Head 2993d737 (tree identical to the readings below; no commit followed them).

skills/** readings — the two required before/after counts (lines and tokens)
File Lines before → after Tokens before → after (ratchet convention)
skills/objectstack-automation/SKILL.md 451 → 451 6466 → 6465 (ceiling re-locked at 6465, headroom 0)
skills/objectstack-automation/references/integration-recipes.md 0 → 150 0 → 2177 (pinned, headroom 0)
skills/objectstack-automation/evals/integrations-declared.json 0 → 35 0 → 1193 (pinned, headroom 0)
whole package (sum of every skills/*/SKILL.md) 4615 → 4615 bundle total (whole shipped tree) 215826 → 219195
Token ratchet and id-lint (exit codes captured before any pipe)
node scripts/check-skills-token-ratchet.mjs --self-test   -> exit 0 ; "✓ check-skills-token-ratchet self-test: 79 cases pass."
node scripts/check-skills-token-ratchet.mjs               -> exit 0 ; "✓ … 84 authored bundle file(s) within their ceilings; 10 generator-owned file(s) measured, not ratcheted."
  skills/objectstack-automation/SKILL.md is 6465 tokens (ceiling 6465; headroom 0)
  skills/objectstack-automation/evals/integrations-declared.json is 1193 tokens (ceiling 1193; headroom 0)
  skills/objectstack-automation/references/integration-recipes.md is 2177 tokens (ceiling 2177; headroom 0)
node scripts/pm/check-skill-id-lint.mjs                   -> exit 0 ; "✓ check-skill-id-lint: 104 file(s) clean"
  probe (a `#26709` line under references/, removed before commit) -> exit 1 ; "✗ … zz-probe.md: 1 issue-ID citation(s)"
Derived gate families — `dispatch-gates --commands` on the actual diff, every command run, reconciled with `--ran`

node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack derived 64 commands for the 5 changed paths; each ran as bash -c CMD > log 2>&1; code=$? and was recorded as CMD :: exit CODE.

node scripts/pm/dispatch-gates.mjs --ran gates-ran-coded.txt --repo objectstack-ai/objectstack -> exit 0: "✓ dispatch-gates --ran: 64 derived famil(ies) accounted for — 59 run, 5 NOT-MEASURED (5 DERIVED from a recorded exit 3)."

Exit 0 (59), among them: check:skill-docs, check:skill-refs (@objectstack/spec), check:doc-authoring, check:nul-bytes, check:pm-skill-id-lint, check:skill-identifier-liveness, check:skill-frame-sync, check:skill-compatibility, check:ratchet-remedy-authority, check:pm-dispatch-gates (412 s), check:changeset-gate-self-tests, check-empty-changeset --base origin/main, check-adr-0087-registration --base origin/main, check-changeset-no-major --base origin/main, check:published-files, check:merge-driver, check:cross-package-test-inputs.

Exit 3 (5, PREREQUISITE NOT MET, NOT MEASURED locally): check:doc-formula-expressions, check:dts-closure, check:dual-build-cjs-loads, check:lean-entry-closure, check:sourcemap-no-sources-content.

Not derived, run on their own: check:skills-token-ratchet family is in the derived list; no packages/spec build was needed (no package source changed, and check:skill-docs / check:skill-refs run from source).

FlowSchema probe for Acceptance note 1
pnpm --filter @objectstack/spec exec tsx scratch-27223-connector-input-probe.ts   (scratch file, deleted before commit)
FlowSchema.safeParse success = true
parsed connectorConfig.input = {"path":{"customer":{"dialect":"cel","source":"record.stripe_customer_id"}}}

CI: the PR's check-runs are in_progress at report time; the PM reads their convergence.


Generated by Claude Code

…on, per-archetype recipes, three evals (WIP)

Co-authored-by: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017xRBcVBKAFk1ntXSjvPt7U
@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

epic:#27293#1 · sunhaolin · session_017xRBcVBKAFk1ntXSjvPt7U

Contract review

Served-tier: CONTRACT_REVIEW_TIER
Head-sha: 2993d7377f2b45cd7fa0d1a72a45405d0507397c
Local-runs: none
Scope: skills/objectstack-automation/SKILL.md, skills/objectstack-automation/evals/integrations-declared.json, skills/objectstack-automation/references/integration-recipes.md + scripts/check-skills-token-ratchet.mjs (the rows priced with SKILL.md), packages/spec/src/shared/connector-auth.zod.ts, packages/spec/src/integration/connector.zod.ts, packages/spec/src/integration/index.ts, packages/spec/src/automation/flow.zod.ts, packages/spec/src/automation/api-trigger-secret-ref.zod.ts, packages/spec/src/automation/schedule-organization.zod.ts, packages/spec/src/kernel/platform-capabilities.ts, packages/spec/src/ui/action.zod.ts, packages/spec/src/data/field.zod.ts, packages/connectors/connector-openapi/src/openapi-provider.ts, packages/connectors/connector-openapi/src/openapi-connector.ts, packages/connectors/connector-mcp/src/mcp-provider.ts, packages/connectors/connector-mcp/src/mcp-connector.ts, packages/connectors/connector-rest/src/rest-provider.ts, packages/connectors/connector-rest/src/rest-connector.ts, packages/triggers/trigger-api/src/api-trigger.ts, packages/triggers/trigger-schedule/src/missed-due-times.ts, packages/services/service-automation/src/builtin/connector-nodes.ts, packages/services/service-automation/src/builtin/value-slot-map.ts, packages/services/service-automation/src/builtin/crud-nodes.ts, packages/services/service-automation/src/dag-traversal.ts, packages/services/service-automation/src/plugin.ts, packages/formula/src/stdlib.ts, packages/create-objectstack/src/templates/blank/objectstack.config.ts, content/docs/automation/connectors.mdx, scripts/pm/check-skill-id-lint.mjs, skills/objectstack-automation/references/examples-flows.md, skills/objectstack-ai/SKILL.md, packages/skills/package.json

Reviewed at 2026-10-28T17:15Z, read-only: the card (body and all seven comments), the PR body, its file list, the net diff against main, and the head's check-runs; every cited file read on origin/main at 19f49ef3, whose cited trees are byte-identical to the PR's merge-base a0aa8a1f (git diff --stat over them is empty), so "the runtime at the PR's base" and main are one reading here. The rest of the diff (.changeset/27223-integrations-declared.md, the non-row lines of scripts/check-skills-token-ratchet.mjs) is the lane seat's ACCEPT at TIER_DEFAULT.

Check-runs on the head (the gate verdicts): 42 runs, all completed — 28 success, 14 skipped, none red, none in progress. The ten required contexts: Lint & Repo Gates, TypeScript Type Check, Test Core, Build Core, Governed Surface Queue Guard, Test UI all success; Lint UI, Type Check UI, Build & E2E UI, Live E2E UI skipped by the path filter (no ui/ path in the diff). Check Changeset, Check PR Size, PR-body guards, Dogfood Regression Gate success.

① Derived judgments

  • Accept-set, connector_action row — "Dispatch an action of a declared connector … or a plugin's. connectorConfig.input is sent as written — no CEL or double-brace holes in it." Right. service-automation/src/builtin/connector-nodes.ts:108 hands cfg.input ?? {} to the handler untouched; builtin/value-slot-map.ts enumerates fields, subflow/map input, script inputs, screen defaults, notify templateData, http body and never a connector slot; flow.zod.ts:950 says the same in the spec's own comment. "Or a plugin's" keeps references/examples-flows.md:49-57 (the slack plugin connector) a legitimate shape.
  • Accept-set, http row — the auth-routing tail ("a header credential → a declarative connector … a path-secret webhook url → a token-authenticated connector") moves to the recipes' "Holds on every recipe" bullet; the row keeps "never a credential there: a credential means a declared connector called via connector_action". Right — nothing taught was dropped, the url/headers-served-as-written warning stays verbatim.
  • Accept-set, the rule section — provider: 'openapi' | 'mcp' | 'rest', auth.credentialRef, connector_action, inbound callback as an api flow. Right: ConnectorBaseSchema.provider/providerConfig/auth (connector.zod.ts:870-905), ConnectorInstanceAuthSchema four variants each with credentialRef (connector-auth.zod.ts:118-163), FlowSchema.type enum carries api (flow.zod.ts:1040).
  • Removals — "pre-built integration connector" gone; "reserve custom code for edge-case integrations" gone; the References section drops the pointer to evals/approvals/test-revise-loop.md and the "ADR-0032 gate" sentence; a stale "(see the approval example below)" pointer gone. Right — trims and the card's two named lines; the graded eval file itself stays in the bundle.
  • Public surface — no packages/* source moves; the catalog ships inside @objectstack/skills dist via packages/skills/scripts/sync-catalog.mjs (its build script), so the bundle gains two files and SKILL.md changes bytes. Right that this is a published-package change and nothing more.
  • Ratchet rows, read with SKILL.md — recomputed from the head's blobs in the gate's own convention: SKILL.md 25880 − 20 stamp bytes = 25860, ceil(/4) = 6465 (base 25864 → 6466); integration-recipes.md 8705 → 2177; integrations-declared.json 4772 → 1193. Right, each pinned at its landed count, headroom 0. Bundle growth is authorized the way the gate's header asks ("a PR whose body quotes a maintainer ruling authorizing it"): the PR body quotes the maintainer's directive that ordered this card verbatim and untranslated — 「所以应该先处理 A/B 吧,并且要有 skills或者文档指导元数据开发者。前面agent还在建议我开发 DocuSign」 and 「hotclm 的开发agent 就搞错了,说明 skills还是不对,要立卡完善」 — and the card's Work items 2 and 3 are the two new files. Right.
  • Eval vocabulary — every must_contain token of the three cases is spelled as a whole token in the head bundle, read rather than run: defineConnector, provider: 'openapi', credentialRef, connector_action, type: 'api', secretRef, Swagger 2.0, blocked, type: 'schedule', runAs: 'system', update_record, dialect: 'cel', operationId, type: 'basic' — all in integration-recipes.md or SKILL.md. The file's shape (skill_name, evals[].id/prompt/expected_output/files/assertions.must_contain/must_not_contain) matches evals/flows-triggers-approvals.json. Right.

② Semver level

@objectstack/skills patch; Clause-②: no; Metadata-protocol: none — on the changeset and the PR body alike. Right: no authorable key, accepted value or metadata type moves (no packages/spec change), no public export moves; the catalog ships in that package's dist, so a changeset is owed and skip-changeset would be wrong. Kind none ⇒ patch. Check Changeset is success on the head.

③ Boundary flags

Every dev flag answered; open_questions is empty.

  1. Card numbers not in the skill text — answered, right: scripts/pm/check-skill-id-lint.mjs forbids #NNN in every .md under skills/ by the maintainer ruling its header quotes (「保留 issue id没有意义,如果ai去查原始issue,得不偿失」, widened to the published catalog later). The ruling outranks the card's Work-item wording; the recipes name each gap and "its platform card", the numbers live in the PR body. No escalation.
  2. references/_index.md untouched — answered, right: generator-owned (gen:skill-refs), outside the ratchet's priced population by the gate's own boundary section; check:skill-refs runs inside Lint & Repo Gates/TypeScript Type Check, both success.
  3. No eval runner in the root repo — true: no pnpm script consumes skills/*/evals/*.json (ui/scripts/check-skill-eval-tokens.mjs scans ui/skills alone). The card's acceptance "pnpm skill evals green" cannot be met as spelled; the dev's scratch whole-token oracle readings are its own and are not evidence this review relies on. Escalated to the epic PM as an acceptance-wording correction (the card, not this PR); the three cases are judged here by reading them against the bundle and the runtime, ④ below.
  4. A third gap in the DocuSign answer (literal connectorConfig.input) and the callback caveat — answered, right; both measured against the executor and the door (④).
  5. More SKILL.md sites moved than the three named — inside the claimed file surface; each read: re-wraps, the Flow Variables example re-spelled, clause trims; no taught key or behaviour lost beyond the two pointers and the ADR-0032 sentence named in ①.
  6. PR-body figure +3,370 vs net +3,369 — cosmetic; the rows are right.
  7. (WIP) in the commit title — cosmetic; the landing squash is the maintainer's hand (Tier H).
  8. Five derived families NOT MEASURED locally — CI measured them after its build: Lint & Repo Gates, Lint · *, Type Check · * all success.

out_of_scope_findings — the class-c trap (a CEL envelope under connectorConfig.input parses success = true and is sent verbatim; the openapi handler then String()s it into the path) is real on the cited code (connector-nodes.ts:108; openapi-connector.ts:388,397) and is exactly Prime Directive #12's "reject at authoring": escalated to the epic PM to file as a card (the dev names no carrier). The skill now states the limit, so no false claim rides on it. The schedule observation is extended in ④.

④ Design judgment

skills/objectstack-automation/SKILL.md — judged against the card's eval prompt (the DocuSign case) and the spec it teaches on main. The rule section sends a vendor-over-HTTP integration to a connectors: entry and connector_action, an inbound callback to an api flow, and names a loader gap as a card, never code; an agent holding the DocuSign prompt and this text lands on recipe 1 + recipe 4 + the "Not yet" OAuth entry and never on a plugin. Every key named is delivered at base (①). The loader-gap list (YAML, Swagger 2.0, $ref, OAuth, request signing) matches openapi-provider.ts (JSON-only parse, assertOpenApi3Document refuses Swagger 2.0 and "never converts"), connector-auth.zod.ts (no oauth2 in the declarative union) and the absence of any signing auth kind. Holds. One should-fix:

  • should-fix (SKILL.md, the five-condition sentence): "Platform code only when one holds: … the callback's signature is outside the api door's HMAC" says platform code is warranted for exactly DocuSign Connect's case, and never says whose code. Read alone, an app-authoring agent can take it as licence for an app plugin — the answer eval 1 refuses (implements Plugin). Recipe 4 ("name the gap and poll its status … until vendor schemes verify") and the eval's expected output resolve it; the rule sentence does not, and the rule is what the agent reads first. Remedy is one clause: platform code is the platform's, filed as a card — the app still writes none (ADR-0019 D2). Not a FAIL: the recipe and the eval pin the DocuSign case, and the card's own Work-item list carries the same five conditions in the same words.

skills/objectstack-automation/references/integration-recipes.md — each connector key, flow key and behaviour checked against the live tree:

  • Recipe 1 (openapi, static key): type: 'api' is in ConnectorTypeSchema; providerConfig.spec as a package-relative path is loaded through ctx.loadPackageFile (openapi-provider.ts:170-200); auth: { type: 'bearer', credentialRef } is the declarative variant; an unresolved ref is a hard boot error (service-automation/src/credential-resolver.ts:6, plugin.ts:2314-2327); type: 'schedule' + runAs: 'system' (flow.zod.ts:1099 — a run with no trigger user must declare system); the start node's config.schedule: { type: 'cron', expression } is what trigger-schedule/src/missed-due-times.ts:64-71 reads; the action key is the operationId (openapi-connector.ts:209); the answer lands as nodeId.status/ok/body (openapi-connector.ts:200, written under node.id.key by dag-traversal.ts:493-497); update_record with a literal filter and CEL-envelope fields is the value-slot shape (value-slot-map.ts, crud-nodes.ts:84-110); isoDatetime(now()) is CEL stdlib (formula/src/stdlib.ts:639); automation/triggers/job are PLATFORM_CAPABILITY_TOKENS. "The openapi provider has no include key" — OpenApiProviderConfig is spec/baseUrl only. Holds, with one should-fix:
    • should-fix (recipe 1, the $ref wording): "(3.x only, package-relative, no $ref into other files)" and "a multi-file $ref tree is converted or bundled offline" narrow the $ref gap to the multi-file case. The loader also leaves an in-document $ref undereferenced: openapi-connector.ts:295 skips a $ref parameter, :342 a $ref requestBody, :336 a $ref response — the action loads and dispatches, but the input/output schema recipe 5 promises ("a named action with its input schema, visible in the flow palette and to agents") silently loses those parts, and a trimmed Stripe document is all component refs. SKILL.md's own gap list says $ref unqualified, so the recipe is the one place that narrows it. Remedy: "dereference every $ref offline, in-document ones included" — one clause. Not a key the runtime lacks and not a dispatch that fails, so not a FAIL.
  • Recipe 2 (mcp, static header): transport: { kind: 'http', url, headers } and include are the provider's keys (mcp-provider.ts:45-48, 108-116); bearer auth is folded into Authorization (:57-70); stdio is denied unless the host allowlists the command (:14-27, 132-149); the answer is { ok, content, structuredContent? } (mcp-connector.ts:163-166); json is a field type. Holds. The GitHub URL and X-MCP-Readonly header are vendor facts outside this review.
  • Recipe 3 (rest): baseUrl and defaultHeaders are the provider's keys (rest-provider.ts:20-21, 42-60); the single request action takes { method, path, headers, query, body } and answers { status, ok, body } (rest-connector.ts:139-192); api-key with headerName is applied per request. The AI action { type: 'flow', target, ai: { exposed: true, description } } is ActionSchema's shape — description ≥ 40 chars is required when exposed (action.zod.ts:789-796), the recipe's is 78; objectstack-ai/SKILL.md § Actions as AI Tools exists. Holds.
  • Recipe 4 (callback): type: 'api', runAs: 'system', start config: { hookId, secretRef } — secretRef is api-trigger-secret-ref.zod.ts's key (env-var-shaped, never OS_), hookId the URL token with default 'default' (api-trigger.ts:217, 258); the path is POST /api/v1/automation/hooks/:flowName/:hookId (:201); the body is bound as the trigger record record.* (:289-293), so record.tracking_no in a CEL filter operand is resolved by resolveNodeFilter; queue is required (:286, 379). The signature paragraph matches the door exactly: timestamped t=unix,v1=hex HMAC-SHA256 of t.rawbody, the body-only sha256= form refused, "no unsigned branch" (:73, 104-113, 141-160). The head's own webhook table (SKILL.md line 351) teaches secretRef the same way. Holds.
  • Recipe 5 (hand-written document): the inline openapi: '3.0.3' object passes assertOpenApi3Document's ^3\.\d+(\.\d+)?$; basic username + credentialRef is the declarative variant; one operationId ⇒ one action. Holds.
  • "Holds on every recipe": actions refused on a provider-bound entry and inline authentication refused — both are DeclarativeConnectorEntrySchema refinements (connector.zod.ts:1168-1216), the schema behind defineStack({ connectors }), so "refused at os validate" is exactly right (note: defineConnector itself parses the plain ConnectorSchema and refuses neither — the recipe's "both register in defineStack" keeps the sentence true). defineConnector reaches @objectstack/spec/integration through integration/index.ts's export * of connector.zod; defineFlow is on the root. "Scaffolded apps list the three executors" — create-objectstack/src/templates/blank/objectstack.config.ts:63-65. The "Not yet" entries name no key: no oauth2, no signing kind, "do not park a short-lived token behind a bearer ref" is the docs page's own sentence. Holds. The administrator credential object is named as "a platform card in flight", no key — Prime Directive chore: version packages #10 honoured.

skills/objectstack-automation/evals/integrations-declared.json — judged against the card's three cases and the bundle:

  • Case 1 (DocuSign): the expected answer is the declarative openapi instance over a trimmed 3.x document converted offline from Swagger 2.0 (the PM's premise correction, right against openapi-provider.ts), an api callback flow with secretRef, three gaps named, marked blocked on OAuth; must_not_contain carries registerConnector(, connector-esign, provider: 'docusign', implements Plugin, clientSecret, token: — any plugin answer fails. Against the base skill the must_contain tokens defineConnector, provider: 'openapi', type: 'api', Swagger 2.0, blocked are spelled nowhere in skills/objectstack-automation/** (the base grep for defineConnector|provider: 'openapi' is the card's premise 2, re-verified 0 files); on the head all eight are taught. Holds. Observation, not a finding: clientSecret in must_not_contain also fails a correct answer that explains the absent oauth2 variant by naming that runtime-shape field; the bundle's own wording avoids it.
  • Case 2 (Stripe read): every element of the expected output is recipe 1's shape, key for key; must_not_contain refuses the http-node-with-header answer. Holds.
  • Case 3 (vendor without spec): recipe 5's shape; provider: 'rest' named as the alternative when the path stays per call. Holds.

Observations outside this PR's claims (no finding, carrier noted): (a) package-authored scheduled work is switched off by default and listed as "disabled by deployment policy" (schedule-organization.zod.ts, the posture table) — the skill teaches this nowhere, before or after this PR, and recipe 1 / case 2 inherit the silence with the catalog's existing scheduled example; a skills card, not this PR. (b) The flow-type table's api row (SKILL.md line 52) still says "needs a start-node secret" while the webhook table teaches secretRef; pre-existing and untouched here.

Attempts to break it that did not land: a CEL envelope inside update_record.filter (recipe 4) — resolved by resolveNodeFilter, not literal; bal.body.available[0].amount — body is written under bal.body by the traversal; requires tokens — all four are in PLATFORM_CAPABILITY_TOKENS; api-key default header — X-API-Key applied in both rest-connector.ts:107 and mcp-provider.ts:77; username in metadata for basic — the declarative variant's own .describe.

Implemented-by: claude/issue-27223-integrations-declared
Reviewed-by: session_017xRBcVBKAFk1ntXSjvPt7U

VERDICT: PASS

Must-fix: none. Should-fix: (1) SKILL.md five-condition sentence — say platform code is the platform's, filed as a card, the app writes none; (2) recipe 1 $ref wording — the loader drops in-document $ref parts from the derived schema too, so "dereference every $ref offline". Both are one-clause edits for the owning seat's next push or the maintainer's landing; neither changes the verdict.

… conditions license, dereference every $ref offline, api row names secretRef, eval token clientSecret:

Co-authored-by: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017xRBcVBKAFk1ntXSjvPt7U
@os-zhuang
os-zhuang marked this pull request as ready for review October 28, 2026 17:49
@os-zhuang
os-zhuang added this pull request to the merge queue Oct 28, 2026
@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

epic:#27293#1 · sunhaolin · session_017xRBcVBKAFk1ntXSjvPt7U

Contract review

Served-tier: CONTRACT_REVIEW_TIER
Head-sha: 51e804554f630c3bd48d3c569eb611dbfd7c6d9c
Local-runs: none
Scope: skills/objectstack-automation/SKILL.md, skills/objectstack-automation/evals/integrations-declared.json, skills/objectstack-automation/references/integration-recipes.md + scripts/check-skills-token-ratchet.mjs (the three rows priced with the skill files), packages/spec/src/automation/api-trigger-secret-ref.zod.ts, packages/triggers/trigger-api/src/api-trigger.ts, packages/connectors/connector-openapi/src/openapi-connector.ts, packages/spec/src/shared/connector-auth.zod.ts

Delta review at 2026-10-28T17:48Z, read-only, of the one commit between the PASS head 2993d737 (record 6475700322) and this head — git diff 2993d737 51e80455, four files, +43/−28 — each hunk judged against the live tree on origin/main 19f49ef3 (its cited trees are byte-identical to the PR's merge-base a0aa8a1f), never against the dev's account of it. Everything outside the delta stands as reviewed in 6475700322.

Check-runs on this head (the gate verdicts): 42 runs, all completed — 28 success, 14 skipped, none red, none pending. The ten required contexts: Lint & Repo Gates, TypeScript Type Check, Test Core, Build Core, Governed Surface Queue Guard, Test UI success; Lint UI, Type Check UI, Build & E2E UI, Live E2E UI skipped by the path filter (no ui/ path).

① Derived judgments

  • Accept-set, flow-type table api row — secret → "needs a start-node secretRef (or a literal secret; see Inbound webhook triggers below)". Right. trigger-api/src/api-trigger.ts:258-275: the arm reads cfg.secret as a literal or takes the engine's resolveSecret (the secretRef path); with neither it throws before anything is stored ("not armed: its start node declares no config.secret … or name the secret in config.secretRef"); api-trigger-secret-ref.zod.ts:127-138 refuses both keys together and a secretRef on a flow with no api trigger. The row now agrees with the head's own webhook table (line 350: "secretRef / secret … one required … Ship secretRef"), which was the inconsistency the first record observed.
  • Accept-set, the rule section — "Platform code only when one holds — and then it is the platform's, filed as a card; the app still writes none: …". Right: the first record's should-fix 1, applied as one clause; ADR-0019 D2 (plugins are internal contributions, never the consumer unit) is the decision it restates, and the DocuSign Connect case now reads the way recipe 4 and eval 1 already resolved it.
  • No other accept-set movement. The remaining SKILL.md hunks are trims that pay the two additions: "(the example apps use plain strings)" dropped from the CEL blockquote (an aside, the rule "pick one per file" stays), "inbox channel" → "inbox", "contributed by" → "from", three "→ Moved to" pointers → "→", one script clause joined, "never in-band" dropped beside "runs on the consumer" (the same fact), the CRM-blueprint intro shortened. Nothing taught is lost.
  • Ratchet rows, read with the skill files — recomputed from this head's blobs in the gate's convention: SKILL.md 25847 − 20 stamp bytes = 25827, ceil(/4) = 6457 (was 6465, −8); integration-recipes.md 8814 → 2204 (was 2177, +27); integrations-declared.json 4773 → 1194 (was 1193, +1). Right — each row re-locked AT its measurement, headroom 0, moved by the measured delta only; the net bundle growth is +20 tokens inside the same card's authorization quoted in the PR body.

② Semver level

Unchanged by the delta: @objectstack/skills patch, Clause-②: no, Metadata-protocol: none on changeset and PR body. Right — the delta touches skill prose, one eval token and three ceilings; no authorable key, export or spec moves. Check Changeset success on this head.

③ Boundary flags

The delta carries no new dev flag and no open_questions; the four claims the coordinator relayed were each judged on the code, not the report:

  1. Should-fix 1 applied — the five-condition sentence now says whose code (①). Closed.
  2. Should-fix 2 applied — recipe 1 now reads "every $ref dereferenced offline" in the comment and "dereference every $ref offline, in-document ones included: the loader skips a $ref parameter, request body or response, and the derived input/output schema silently loses it" in the prose. Right against openapi-connector.ts:295 (a $ref parameter is skipped), :342 (a $ref requestBody yields no body schema), :336 (a $ref response yields no output schema). Closed.
  3. api row fixed in place — right (①); done inside the already-claimed file surface, no new surface.
  4. Eval 1 must_not_contain clientSecret → clientSecret:, token: kept — right: a correct answer that explains the absent oauth2 variant by naming clientSecret as a field of the runtime shape (connector-auth.zod.ts:35) now passes, while an answer writing the runtime literal clientSecret: '…' — the plugin or ConnectorOAuth2Schema path — still fails; token: still refuses an inline bearer. The first record's observation, closed. No must_contain token changed, so the bundle still spells every one.

Escalations from the first record stand unchanged (the card's "pnpm skill evals" acceptance wording; the class-c connectorConfig.input envelope trap to file as a card; the deployment-policy switch for package-authored schedules, a skills card). None is this PR's to carry.

④ Design judgment

skills/objectstack-automation/SKILL.md — against the card's DocuSign eval prompt: with the new clause an agent reading only the rule section can no longer take "the callback's signature is outside the api door's HMAC" as licence for an app plugin — the sentence itself now sends it to a card, which is the answer eval 1's must_not_contain accepts and recipe 4 teaches. The api row names the key the recipes and the webhook table use (secretRef) and keeps the literal secret as the door's other accepted form, exactly as api-trigger.ts arms it. The trims change no taught key or behaviour. Holds.

skills/objectstack-automation/references/integration-recipes.md — recipe 1's loader paragraph is now exact against the loader: Swagger 2.0 / YAML converted offline (openapi-provider.ts parses JSON only and assertOpenApi3Document refuses 2.0, "never converts"), every $ref dereferenced offline because the generator skips a $ref parameter, request body or response (openapi-connector.ts:295,336,342), no include key (OpenApiProviderConfig is spec/baseUrl). The joined "Full guide" line is a re-wrap. No key the runtime does not deliver. Holds.

skills/objectstack-automation/evals/integrations-declared.json — one token: clientSecret:. The case still fails any plugin answer (registerConnector(, connector-esign, provider: 'docusign', implements Plugin) and any inline secret (clientSecret:, token:), and no longer penalises the correct explanation of the absent variant. Holds.

Tried to break it: an api flow with secretRef alone — armed through the engine's resolver (api-trigger.ts:259, 274), so "(or a literal secret)" is the right parenthesis and not the lead; a flow with both — refused at parse (api-trigger-secret-ref.zod.ts:127); an in-document $ref document — loads and dispatches, schema lossy, which is now what the recipe says; the ratchet rows — each equals its blob's ceil(bytes/4).

Must-fix: none. Should-fix: none.

Implemented-by: claude/issue-27223-integrations-declared
Reviewed-by: session_017xRBcVBKAFk1ntXSjvPt7U

VERDICT: PASS

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

epic:#27293#1 · sunhaolin · session_017xRBcVBKAFk1ntXSjvPt7U

维护者速读

  • 改了什么:objectstack-automation 技能新增「Integrations are declared, not coded」一节(厂商 HTTP 系统 = 应用内声明的 connectors: 条目 + connector_action,回调走 api 流;只有五种情形才需要平台代码,而且那是平台的卡,应用本身仍不写代码);connector_action 行去掉 "pre-built";新增六种原型配方 references/integration-recipes.md 和三个评测例 evals/integrations-declared.json(DocuSign 例对任何「写插件」的答案判失败)。
  • 为什么改:HotCLM 的开发 agent 读了技能后要求平台造 DocuSign 插件(你的原话「hotclm 的开发agent 就搞错了,说明 skills还是不对,要立卡完善」)。
  • 风险与回滚:只改技能文本、评测与棘轮行,无运行时改动;SKILL.md 净 −9 token(不是正文写的 −1),整包净 +3,389 token(不是 +3,370);回滚即 revert。
  • 先看:SKILL.md 新节的「五种情形」句;配方「Holds on every recipe」里 connectorConfig.input 原样发送那条(今天逐记录取值不能声明,已立 automation: a connector_action node's connectorConfig.input is handed to the provider as written — a CEL envelope there passes os validate and is never evaluated, so a flow cannot send a per-record value to a declarative connector #27327)。
  • 两分钟验证:node scripts/check-skills-token-ratchet.mjs 预期 ✓(SKILL.md 6457/6457、配方 2204/2204、评测 1194/1194)。
  • 需要你做的:无 —— 你已在 17:49Z 亲手翻 ready 并入队,这就是本 PR 的人工审核记录。

席位意见: 赞成合入。两轮达档契约复核 PASS(6475700322 @ 2993d737,增量 6476424433 @ 51e80455),首轮两条 should-fix 与两处顺带修正已在 51e80455 落实。配方只教 base 上已交付的键;include(openapi)、oauth2、signed 以「尚未支持、等平台卡」的形式出现。

席位核对

ACCEPT at head 51e80455 · Implemented-by: claude/issue-27223-integrations-declared · Reviewed-by: this seat (lane ACCEPT at TIER_DEFAULT) + the isolated CONTRACT_REVIEW_TIER reviewer (records above).

Item Reading
Shape first line Fixes #27223; base main; Tier H (skills/**), so the maintainer's hand landed it: os-zhuang ready_for_review + added_to_merge_queue at 2026-10-28T17:49Z
Premise, measured holds on origin/main (pre-built integration connector at SKILL.md:89 on base; no reference or eval covered a provider-bound instance); the claim's correction (allowlist = trimmed packaged 3.x document, include not delivered on openapi) is what the eval teaches
Gates (check-runs) 51e80455: 42 runs, 28 success, 14 skipped (UI path filter and opt-in), none red; required Lint & Repo Gates, TypeScript Type Check, Test Core, Build Core, Governed Surface Queue Guard success
Accepted deviations card numbers kept out of skill text (check-skill-id-lint ruling outranks the card's wording); references/_index.md untouched (generator-owned); SKILL.md api row secret → secretRef (or a literal secret), an in-place fix in the claimed file, verified at api-trigger.ts:258-275; the PR body's token figures are stale (corrected in the brief above; the dev writes the body once)
Out of scope filed #27327 (connectorConfig.input sent as written; a CEL envelope there validates); Acceptance notes: the card's "pnpm skill evals green" has no runner in the root repo (corrected on #27223); in-document $ref skipped by the openapi loader without a diagnostic → carrier #26709 (normalization item); the package-schedule deployment-policy switch taught nowhere in the skill, and the pre-existing evals 7/8 tokens → noted, not filed. Skills-docs: n/a (Metadata-protocol: none)
Landing in the merge queue by the maintainer; the seat follows it to MERGED and closes the card's state labels

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/m tooling

Projects

None yet

3 participants