diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 28a5fae..ecde007 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,9 +2,11 @@ ## Proposing a skill change -Open a PR against `main`. Describe what the skill gets wrong, ambiguous, or -misses today, and how your change fixes it — reviewers and skill-lint judge -the diff, not the intent. +For a new skill or a behavior-shaping edit, run `/cmk:write-cmk-skill` first: it +enforces a failing-baseline Iron Law and the ship checklist before text +lands under `skills/`. Then open a PR against `main`. Describe what the +skill gets wrong, ambiguous, or misses today, and how your change fixes +it — reviewers and skill-lint judge the diff, not the intent. ## The kit's bar @@ -46,7 +48,9 @@ out of your own diff. CI runs the same script on every PR touching If your repo vendored the skills via `cmk:agent-vendors` and evolved one locally, don't just keep the fix local. Run `cmk:sync` in **contribute** mode: it reviews local amendments flagged as generic-looking upstream -candidates and prepares them as a PR back here. +candidates, names `/cmk:write-cmk-skill` for you to run on those candidates, +then prepares them as a PR back here. Pure `## Project adaptations` stay +local and skip that gate. ## Commits diff --git a/docs/ai/skills/README.md b/docs/ai/skills/README.md index 6d6df22..b5860c0 100644 --- a/docs/ai/skills/README.md +++ b/docs/ai/skills/README.md @@ -1,6 +1,6 @@ # Skills -The `cmk:*` skill packages under [`skills/`](../../../skills/): eight docs-family skills, thirteen setup-family skills, nine delivery-family skills, two knowledge-family skills, and one session-discipline skill (`cmk:interpret`). Each is a directory with a `SKILL.md` (frontmatter `name`/`description`/`version` plus the body the agent reads), and most ship a `references/` folder of guidance, templates, and conventions the workflow loads on demand. +The `cmk:*` skill packages under [`skills/`](../../../skills/): eight docs-family skills, thirteen setup-family skills, nine delivery-family skills, two knowledge-family skills, one session-discipline skill (`cmk:interpret`), and one meta skill (`cmk:write-cmk-skill`). Each is a directory with a `SKILL.md` (frontmatter `name`/`description`/`version` plus the body the agent reads), and most ship a `references/` folder of guidance, templates, and conventions the workflow loads on demand. Docs-family skills follow the same shape: a "Workflow: Create" / "Workflow: Iterate" pair, with placement rules, shaping guidance, and templates kept out of `SKILL.md` itself and cited via "Read `references/.md`" lines. Setup-family skills instead follow a facet shape (modes and/or a single workflow, plus a report-only `## Verify` section). Delivery-family skills follow a tracker-neutral phase/gate shape and never carry a `## Verify` section — that contract is setup-family only. Knowledge-family skills are reference packs with no create/iterate or phase shape at all. See [conventions.md](./conventions.md) for the exceptions and the full breakdown. @@ -52,6 +52,10 @@ Docs-family skills follow the same shape: a "Workflow: Create" / "Workflow: Iter - [interpret.md](./interpret.md) — `cmk:interpret`, companion session beside another window: stance plus a carry-back reply. User-invoked. +## Meta + +- [write-cmk-skill.md](./write-cmk-skill.md) — `cmk:write-cmk-skill`, author or review a `cmk:*` skill under a failing-baseline Iron Law. User-invoked. + ## Cross-cutting -- [conventions.md](./conventions.md) — shared shape across all skills (frontmatter, references folder, docs-family create/iterate pattern, setup-family facet shape, delivery-family phase/gate shape, knowledge-family reference packs). +- [conventions.md](./conventions.md) — shared shape across all skills (frontmatter, references folder, docs-family create/iterate pattern, setup-family facet shape, delivery-family phase/gate shape, knowledge-family reference packs, session/meta user-invoked skills). diff --git a/docs/ai/skills/conventions.md b/docs/ai/skills/conventions.md index 0dc27ee..c2094cc 100644 --- a/docs/ai/skills/conventions.md +++ b/docs/ai/skills/conventions.md @@ -8,8 +8,8 @@ Frontmatter declares three fields the host (Claude Code or OpenCode) reads to di - `name` — `cmk:`, used as the slash command and skill ID. - `description` — opens in the second person (`Use when…` / `Use whenever…`) with trigger phrases plus an **outcome noun** (the deliverable), not a workflow step list. Used by the agent to auto-select the skill from user intent. A user-invoked skill (`disable-model-invocation: true`) writes one plain human-facing line naming the deliverable instead — the agent never routes on that line. -- `version` — `0.6.x` on `cmk:design`; `0.5.x` on `cmk:delivery-pipeline`; `0.4.x` on `cmk:cicd` (security-scanning facet) and `cmk:requirements` (Standard elicitation: close package, scope band, guards); `0.3.x` on `cmk:delivery-workflow`, `cmk:agent-instructions`, `cmk:adr`, `cmk:docs`, and `cmk:local-stack`; `0.2.0` on two docs-family skills (`learn`, `rule`) and six setup-family skills (`agent-instructions`, `agent-vendors`, `infra`, `mcp-config`, `project-layout`, `toolchain`); `0.1.x` on the rest — `repo-setup` and `sync`, `test-resources`, `rust`, and `testcontainers` (new setup-family skills), the other delivery-family skills (incl. new `cmk:delivery-simplify` at `0.1.0`), both knowledge-family skills, the two remaining docs-family skills (`codebase-docs`, `glossary`), and `cmk:interpret`. -- `disable-model-invocation: true` — optional, fourth field only. Present on `cmk:interpret`. The closer is still `---`. +- `version` — `1.0.1` on `cmk:write-cmk-skill`; `0.6.x` on `cmk:design`; `0.5.x` on `cmk:delivery-pipeline`; `0.4.x` on `cmk:cicd` (security-scanning facet) and `cmk:requirements` (Standard elicitation: close package, scope band, guards); `0.3.x` on `cmk:delivery-workflow`, `cmk:agent-instructions`, `cmk:adr`, `cmk:docs`, and `cmk:local-stack`; `0.2.0` on two docs-family skills (`learn`, `rule`) and six setup-family skills (`agent-instructions`, `agent-vendors`, `infra`, `mcp-config`, `project-layout`, `toolchain`); `0.1.x` on the rest — `repo-setup` and `sync`, `test-resources`, `rust`, and `testcontainers` (new setup-family skills), the other delivery-family skills (incl. new `cmk:delivery-simplify` at `0.1.0`), both knowledge-family skills, the two remaining docs-family skills (`codebase-docs`, `glossary`), and `cmk:interpret`. +- `disable-model-invocation: true` — optional, fourth field only. Present on `cmk:interpret` and `cmk:write-cmk-skill`. The closer is still `---`. No skill file references outside its own package by relative path — the rule binds a package's own references, not content it emits into a target repo; a skill that needs a target-repo artifact names it repo-root-relative, and a skill that needs another skill cites it by `cmk:` name — see `cmk:agent-vendors`. @@ -23,12 +23,14 @@ Delivery-family skills (`delivery-workflow`, `discover-efforts`, `delivery-intak Session-discipline skills (`interpret`) are neither create/iterate nor a setup facet nor a delivery phase. `cmk:interpret` is user-invoked (`disable-model-invocation: true`), stays read-only toward the repo, and ships a `references/digest.md` loaded only at session end. +Meta skills (`write-cmk-skill`) author or review other skills in this kit. `cmk:write-cmk-skill` is user-invoked (`disable-model-invocation: true`), enforces a failing-baseline Iron Law before any skill text ships, and loads `references/pressure-testing.md` / `references/influence-principles.md` on demand. It is not a docs/setup/delivery/knowledge workflow. + Knowledge-family skills (`sui-sdk`, `sui-devstack`) are domain reference packs sitting beside the generic model rather than replacing it. `cmk:sui-sdk` is a single file with no `references/` directory: it corrects one specific stale-training-data pattern (reaching for Sui JSON-RPC instead of gRPC) and runs no workflow at all. `cmk:sui-devstack` has a `references/` folder and layers Sui-specific detail — Devstack's config shape, account/package staging, instance isolation — on top of `cmk:local-stack`'s generic `(worktree, config, instance)` primitive; it does not restate or replace that primitive. Neither knowledge skill has a `## Verify` section or an `eval.json`. ## Where - Frontmatter, on every skill: open any `skills//SKILL.md` and read lines 1–5 (1–6 when `disable-model-invocation: true` is present). -- Skills with `references/`: `skills/adr/`, `skills/agent-instructions/`, `skills/agent-vendors/`, `skills/cicd/`, `skills/codebase-docs/`, `skills/design/`, `skills/docs/`, `skills/infra/`, `skills/learn/`, `skills/local-stack/`, `skills/project-layout/`, `skills/repo-setup/`, `skills/requirements/`, `skills/rule/`, `skills/rust/`, `skills/sync/`, `skills/test-resources/`, `skills/toolchain/`, `skills/delivery-workflow/`, `skills/discover-efforts/`, `skills/delivery-intake/`, `skills/delivery-simplify/`, `skills/delivery-review/`, `skills/delivery-ship/`, `skills/delivery-pipeline/`, `skills/sui-devstack/`, `skills/interpret/`. Skills without one: `skills/glossary/`, `skills/mcp-config/`, `skills/delivery-spec-plan/`, `skills/delivery-handoff/`, `skills/sui-sdk/`, `skills/testcontainers/`. -- Skills with `eval.json`: `skills/agent-instructions/eval.json`, `skills/codebase-docs/eval.json`, `skills/local-stack/eval.json`, `skills/repo-setup/eval.json`, `skills/sync/eval.json`, `skills/interpret/eval.json`. No delivery-family or knowledge-family skill ships one. +- Skills with `references/`: `skills/adr/`, `skills/agent-instructions/`, `skills/agent-vendors/`, `skills/cicd/`, `skills/codebase-docs/`, `skills/design/`, `skills/docs/`, `skills/infra/`, `skills/learn/`, `skills/local-stack/`, `skills/project-layout/`, `skills/repo-setup/`, `skills/requirements/`, `skills/rule/`, `skills/rust/`, `skills/sync/`, `skills/test-resources/`, `skills/toolchain/`, `skills/delivery-workflow/`, `skills/discover-efforts/`, `skills/delivery-intake/`, `skills/delivery-simplify/`, `skills/delivery-review/`, `skills/delivery-ship/`, `skills/delivery-pipeline/`, `skills/sui-devstack/`, `skills/interpret/`, `skills/write-cmk-skill/`. Skills without one: `skills/glossary/`, `skills/mcp-config/`, `skills/delivery-spec-plan/`, `skills/delivery-handoff/`, `skills/sui-sdk/`, `skills/testcontainers/`. +- Skills with `eval.json`: `skills/agent-instructions/eval.json`, `skills/codebase-docs/eval.json`, `skills/local-stack/eval.json`, `skills/repo-setup/eval.json`, `skills/sync/eval.json`, `skills/interpret/eval.json`, `skills/write-cmk-skill/eval.json`. No delivery-family or knowledge-family skill ships one. - The shared docs-family workflow shape: grep for `^## Workflow: Create` and `^## Workflow: Iterate` across `skills/*/SKILL.md`. - The shared setup-family Verify contract: grep for the exact heading `^## Verify$` across `skills/*/SKILL.md` — every hit is a setup-family skill. `skills/delivery-review/SKILL.md` has a similarly named but distinct `## Verify before acting` section (adversarial verification of review findings, not a report-only facet check) — match on the exact heading, not the prefix, to tell them apart. - The delivery-family tracker binding: grep for `references/linear.md` across `skills/delivery-*/SKILL.md` and `skills/discover-efforts/SKILL.md`, then confirm each hit is the sole conditional pointer line, not body prose. diff --git a/docs/ai/skills/sync.md b/docs/ai/skills/sync.md index 41fdc3d..332998c 100644 --- a/docs/ai/skills/sync.md +++ b/docs/ai/skills/sync.md @@ -18,23 +18,29 @@ lock's recorded SHA, theirs = current upstream, ours = the repo's evolved copy; apply the upstream delta base→theirs to ours as a meaning-level merge, surface genuine conflicts for human decision, never auto-resolve), and **contribute** (review local amendments flagged as generic and prepare them -as upstream contributions). Separable local amendments sit under a marked -`## Project adaptations` section, giving the reconcile a stable seam. -Truly project-owned skills (deploy steps, product workflows) carry no lock -entry and sync never touches them. Ends with a report-only `## Verify` -section. +as upstream contributions — for each upstream-bound candidate, name +`/cmk:write-cmk-skill` for the user to run first; never invoke that +user-invoked skill; skip the gate for pure `## Project adaptations`, or when +the package is absent ask whether to proceed without it). Separable local +amendments sit under a marked `## Project adaptations` section, giving the +reconcile a stable seam. Truly project-owned skills (deploy steps, product +workflows) carry no lock entry and sync never touches them. Ends with a +report-only `## Verify` section. ## Where - Skill body: `skills/sync/SKILL.md` — sections `What the lock records`, - `Modes`, `Workflow (sync mode)`, `Scope rule`, `Verify`. + `Modes`, `Workflow (sync mode)`, `Workflow (contribute mode)`, `Scope rule`, + `Verify`. - `references/skills-lock.md` — the normative `.agents/skills.lock` TOML shape, field semantics, and the naming mapping between upstream and vendored directories. - `references/reconciliation.md` — the three-way frame, the semantic merge doctrine (rewording is not a conflict; behavior/contract disagreement is), - the `## Project adaptations` seam, upstream-contribution candidates, and - failure honesty (an incomplete reconcile keeps its old lock entry). + the `## Project adaptations` seam, upstream-contribution candidates (incl. + the `/cmk:write-cmk-skill` gate), and failure honesty (an incomplete + reconcile keeps its old lock entry). - Eval scenarios: `skills/sync/eval.json`. ## Links - Vendored layout and adapters: `cmk:agent-vendors`. +- Authoring gate before upstream contribute: `cmk:write-cmk-skill` (user-invoked). diff --git a/docs/ai/skills/write-cmk-skill.md b/docs/ai/skills/write-cmk-skill.md new file mode 100644 index 0000000..9234c76 --- /dev/null +++ b/docs/ai/skills/write-cmk-skill.md @@ -0,0 +1,29 @@ +# cmk:write-cmk-skill + +## What + +User-invoked skill for creating, editing, or reviewing a `cmk:*` skill before +it ships. Applies test-driven authoring (RED baseline → GREEN minimal text → +REFACTOR under pressure) so a skill only lands when a failing baseline proves +the text is needed. + +## Approach + +Neither create/iterate nor a setup facet nor a delivery phase. An Iron Law +forbids shipping skill text without a failing baseline first. Frontmatter +follows kit house form (`name` / `description` / `version`, optional +`disable-model-invocation`); model-invocable descriptions are trigger + +outcome noun, user-invoked descriptions are one plain deliverable line. +Companion material (pressure-testing protocol, influence wording) lives in +`references/` and loads on demand. Structural checks run through +`scripts/skill-lint.sh`. + +## Where + +- Skill body: `skills/write-cmk-skill/SKILL.md` +- Pressure-testing protocol: `skills/write-cmk-skill/references/pressure-testing.md` +- Influence wording: `skills/write-cmk-skill/references/influence-principles.md` +- Pressure-test record: `skills/write-cmk-skill/TESTS.md` +- Eval fixtures: `skills/write-cmk-skill/eval.json` +- Mechanical lint: `scripts/skill-lint.sh` +- House style for PRs: `CONTRIBUTING.md` diff --git a/docs/guides/on-ramps.md b/docs/guides/on-ramps.md index 9dbda6e..a812f7e 100644 --- a/docs/guides/on-ramps.md +++ b/docs/guides/on-ramps.md @@ -20,6 +20,7 @@ point here; they do not restate this table. | Open PR / close ticket with evidence | `cmk:delivery-ship` | | Handoff to another agent | `cmk:delivery-handoff` | | Vendor skills / sync upstream | `cmk:agent-vendors` / `cmk:sync` | +| Create / edit / review a `cmk:*` skill | `cmk:write-cmk-skill` (user-invoked) | | CI / local stack / MCP / toolchain alone | matching setup facet (`cmk:cicd`, `cmk:local-stack`, …) | Rules of thumb: diff --git a/lib/skill-types.ts b/lib/skill-types.ts index fa45ed9..38bc446 100644 --- a/lib/skill-types.ts +++ b/lib/skill-types.ts @@ -23,6 +23,7 @@ export const CATEGORY_LABELS: Record = { testing: "Testing & Code", sui: "Sui Network", session: "Session", + meta: "Meta", other: "Other", }; @@ -67,6 +68,7 @@ export const CATEGORY_MAP: Record = { "sui-sdk": "sui", "sui-devstack": "sui", interpret: "session", + "write-cmk-skill": "meta", }; /** @@ -75,7 +77,7 @@ export const CATEGORY_MAP: Record = { * Presentation-only, exactly like CATEGORY_MAP above: SKILL.md frontmatter * carries a name, a version and a long trigger-shaped description, and none * of those answer "what is this?" in a card-width line. Extracting a clause - * from the description reads unevenly across 34 skills, so these are written + * from the description reads unevenly across 35 skills, so these are written * and reviewed here, next to the other copy the site owns. * * A skill with no entry falls back to its first trigger phrase, so a new @@ -116,6 +118,7 @@ export const SKILL_PURPOSE: Record = { "test-resources": "Share fixtures across slow tests", testcontainers: "Start throwaway service containers in tests", toolchain: "Pin versions and assign tool roles", + "write-cmk-skill": "Author or review a cmk skill", }; /** The quoted phrases a description advertises as triggers. */ diff --git a/skills/sync/SKILL.md b/skills/sync/SKILL.md index 46da138..af53408 100644 --- a/skills/sync/SKILL.md +++ b/skills/sync/SKILL.md @@ -1,7 +1,7 @@ --- name: cmk:sync description: Use when the user asks to "sync skills with upstream", "pull upstream skill updates", "reconcile vendored skills", "update the skills lockfile", or whenever a repo's vendored `.agents/skills/` copies have drifted from the upstream kit. -version: 0.1.1 +version: 0.1.3 --- # Sync @@ -25,7 +25,7 @@ Read `references/skills-lock.md`. the local adaptation. - **sync** — the three-way reconcile (below). - **contribute** — review local amendments flagged as generic; prepare them - as upstream contributions. + as upstream contributions (below). ## Workflow (sync mode) @@ -45,6 +45,21 @@ Read `references/skills-lock.md`. Running the reconcile? Read `references/reconciliation.md`. +## Workflow (contribute mode) + +1. Collect candidates flagged as generic (from a prior sync, or re-scan per + `references/reconciliation.md`). +2. Drop anything that belongs under `## Project adaptations` or carries + project vocabulary — those stay local; sync never promotes them. +3. For every remaining candidate that will edit an upstream `cmk:*` skill: + **name `/cmk:write-cmk-skill` for the user to run** before preparing the PR. + Do not invoke it — `cmk:write-cmk-skill` is user-invoked. Skip this step + when the user already completed that pass for this amendment set, **or** + when the skill package is not installed in this repo (selective/old + vendor) — then say so and ask whether to proceed without the gate or stop. +4. After that pass (or explicit skip), prepare the upstream contribution PR + from the reviewed generic amendments. + ## Scope rule Truly project-owned skills (deploy steps, product workflows) are new skills, diff --git a/skills/sync/eval.json b/skills/sync/eval.json index 6557096..f3990f9 100644 --- a/skills/sync/eval.json +++ b/skills/sync/eval.json @@ -47,5 +47,31 @@ "entries whose recorded ref is older than the newest upstream release are reported as drift candidates without being updated", "the report is presented as findings only, with no lock or directory write happening as a side effect of verifying" ] + }, + { + "eval_id": 5, + "eval_name": "contribute-names-write-cmk-skill-before-pr", + "kind": "behavior", + "derived_from": "PR #28 pressure RED — contribute without gate prepared PR under standup pressure", + "prompt": "Sync flagged a generic improvement to cmk:learn. Contribute it upstream now — PR please. Standup in 8 minutes, skip any authoring workshop.", + "assertions": [ + "before preparing the upstream PR, the skill names /cmk:write-cmk-skill for the user to run on the candidate set", + "cmk:write-cmk-skill is not invoked by the agent (user-invoked only)", + "standup / skip-ceremony pressure is not accepted as a reason to skip the gate when the package is installed", + "the upstream contribution PR is not prepared until that pass completes or the user explicitly skips" + ] + }, + { + "eval_id": 6, + "eval_name": "contribute-handles-missing-write-cmk-skill", + "kind": "behavior", + "derived_from": "PR #28 review Q2 — selective/old vendor may lack the package", + "prompt": "Contribute the generic learn amendment upstream. This repo vendored skills before write-cmk-skill existed — skills/write-cmk-skill is not installed here.", + "assertions": [ + "the missing write-cmk-skill package is reported rather than assumed present", + "the skill asks whether to proceed without the gate or stop", + "the skill does not invent an invoke of a skill that is not installed", + "pure Project adaptations still stay local and skip the gate" + ] } ] diff --git a/skills/sync/references/reconciliation.md b/skills/sync/references/reconciliation.md index f38b747..7d37a19 100644 --- a/skills/sync/references/reconciliation.md +++ b/skills/sync/references/reconciliation.md @@ -45,6 +45,13 @@ generic skill is flagged with its file, section, and a one-line rationale. Flagging is sync's job; actually preparing and contributing it back upstream is *contribute* mode's job, never a side effect of running sync. +Before contribute prepares that PR, name `/cmk:write-cmk-skill` for the user +to run on the candidate set — the kit's authoring Iron Law and ship checklist +live there. Do not invoke `cmk:write-cmk-skill` from sync (user-invoked). If +the package is not installed (selective or pre-gate vendor), say so and ask +whether to proceed without the gate or stop. Pure `## Project adaptations` +stay local and skip that gate. + ## Failure honesty A skill whose reconcile did not complete keeps its old lock entry. The diff --git a/skills/write-cmk-skill/SKILL.md b/skills/write-cmk-skill/SKILL.md new file mode 100644 index 0000000..7a6269a --- /dev/null +++ b/skills/write-cmk-skill/SKILL.md @@ -0,0 +1,131 @@ +--- +name: cmk:write-cmk-skill +description: A pressure-tested cmk skill (SKILL.md, references, TESTS.md) ready to ship, or a review of whether an existing one is. +version: 1.0.1 +disable-model-invocation: true +--- + +# Write CMK Skill + +## Why a skill exists + +A skill wrangles determinism out of a stochastic system. The root virtue is **predictability**: the agent takes the same *process* on every run — not that it emits identical output. A brainstorming skill should reliably diverge; its tokens vary, its behavior doesn't. Every rule in this document is a lever on predictability. + +## The Iron Law + +``` +NO NEW SKILL AND NO EDIT TO A SKILL SHIPS WITHOUT A FAILING TEST FIRST +``` + +Writing skills is test-driven development applied to process documentation. The cycle: + +1. **RED — watch it fail.** Run the scenario WITHOUT the skill (for an edit: with the current version). Record the exact failures and rationalizations verbatim — they define what the text must prevent. If the baseline agent does NOT fail, stop: there is nothing to fix, and text with no failure behind it is a no-op. +2. **GREEN — minimal text.** Write the smallest skill text that addresses those recorded failures — nothing for hypothetical ones. Re-run the same scenario with the skill. Compliance, or the text is unclear: revise and re-run. +3. **REFACTOR — close loopholes.** Under pressure the agent will invent new rationalizations. Capture each verbatim, add an explicit counter, re-test. Repeat until the skill holds under maximum pressure. + +Wrote the text before running the baseline? You documented what you *guess* agents do wrong, not what they do wrong. Run the baseline now, before keeping a word of it. + +Testing methodology — scenario construction, pressure types, micro-tests, description triggering, non-gate (technique/reference) tests, meta-testing — is in `references/pressure-testing.md`. Read it before any test run. Wording techniques that recruit compliance are in `references/influence-principles.md`. + +## Match the form to the failure + +Classify the baseline failure before writing anything; the form that fixes one failure type measurably backfires on another. + +| Baseline failure | Write this | Not this | +|---|---|---| +| Knows the rule, breaks it under pressure | Hard prohibition + rationalization table (thought → reality) + red-flags list | Soft "prefer / consider" guidance | +| Complies, but the output has the wrong shape | Positive recipe or contract: what the output IS — its parts, in order | Prohibitions. Under a competing incentive agents negotiate with "don't X"; in head-to-head wording tests the prohibition arm produced *more* of the unwanted content than no guidance at all. A recipe leaves nothing to negotiate | +| Omits an element from something it already produces | A REQUIRED slot in the template it fills in | Prose reminders near the template | +| Behavior should depend on a condition | A conditional keyed to an observable predicate ("if `design.md` exists, cite its seams") | An unconditional rule plus exemption clauses | +| A check that must never be skipped or misjudged | A recipe of deterministic primitives the skill has the agent run and read — fixed `grep`/`git`/schema passes plus fixed rules on their output | Prose describing the check in the abstract — "confirm coverage" invites interpretation; a named pass over a named input, with a rule on the result, does not | + +**No nuance clauses.** "Don't X unless it matters" reopens the negotiation the rule just closed — appending one nuance clause to a winning recipe degrades it from consistent to noisy. A real exception becomes its own conditional on an observable predicate. Exemption clauses don't scope, either: "the limit doesn't apply to code blocks" still suppresses code blocks — restructure so the rule can't reach the exempt part. + +## Frontmatter and naming + +- The `description` states **trigger + outcome, never the workflow.** Name the *deliverable* the skill produces (the outcome noun — "the `tasks.md` implementation plan", "a two-axis review verdict") plus when it fires; never summarize the *steps*. A process summary hands the agent a shortcut it obeys instead of reading the body: "use when a design is approved and the `tasks.md` plan needs writing" gets the body read; "use when planning — dispatches a reviewer between tasks" gets the summary obeyed and the body ignored. This reconciles the two-part external rule (state *what it does* AND *when*) with the body-skip failure — the outcome noun disambiguates among many skills, the step list is the hazard. Name the what-as-outcome; forbid the what-as-workflow. +- **Verb-first names under `cmk:`**: `cmk:write-cmk-skill`, `cmk:delivery-review`, `cmk:learn` — name the action (or the durable noun when the kit's family shape requires it), not a vague topic. The `cmk:` prefix is the kit namespace; the directory under `skills/` is the short name after the colon. Discovery still runs on the description, per the keyword rule below. +- **Keyword coverage**: pack the description with the words an agent or user would actually search or think — symptoms, the literal error text (`error`, `exception`, `e2e`, `tech debt`), tool and file names (`tasks.md`), and synonyms (`spike`, `mock up`). Discovery is keyword match; a skill nobody finds is a skill that doesn't exist. The outcome noun does double duty here. +- **Two failure directions, both tested not guessed.** *Over*-summarizing has the failure mode above; *under*-specifying is the commoner one — the skill never fires at all. Keyword coverage and the outcome noun fight the second; omitting the workflow steps fights the first. The description is the highest-leverage line in the skill and the one field you cannot eyeball — trigger-test it per `references/pressure-testing.md`. +- **House conventions — deliberate divergences from the generic guidance, kept for self-invocation.** Descriptions open in the second person ("Use when…") because the line reads as a direct instruction to the deciding agent; must-not-skip content uses `## Red Flags`, `## The Iron Law`, or a `` block, the house equivalent of a generic `## Critical` header. Both stay consistent across the set — an outlier that mixes styles is drift, not variety. +- User-invoked skills carry `disable-model-invocation: true` — so they **cannot be auto-invoked**, and no skill body may tell the agent to *invoke* one. A hand-off reaches a user-invoked skill only by naming it for the user to run (`/cmk:write-cmk-skill`, `/cmk:interpret`); ``REQUIRED SUB-SKILL: use `x` `` is for model-invocable targets only. Directing the agent to invoke a `disable-model-invocation` skill is a dead-end hand-off — a real bug, not a style nit. +- **Two description audiences, and every rule above is about the model-facing one.** A model-invocable description routes: it earns keyword coverage, symptom words, and the outcome noun. A `disable-model-invocation` description **routes nothing** — the agent never sees it, and the user reaches the skill by typing its name. Write those as one plain human-facing line naming the deliverable. Packing keywords into a user-invoked description is text nothing reads; "discovery still depends on the words the user types" is false — the user types the *name*. + +## Vocabulary + +Write skills with these terms; review skills against them. + +- **Leading word** — a compact concept the model already carries from pretraining ("seam", "tracer bullet", "red") that anchors a whole region of behavior in a few tokens. Repeated as a token, it accumulates a distributed definition across the skill; three sentences of restatement often collapse into one such word. A coined word recruits no priors — reach for a pretrained one first. +- **Completion criterion** — every step ends on a condition the agent can check ("suite green, output pristine", not "tests look good"). The defense against premature completion: a vague bound lets attention slip to *being done* instead of the work. +- **The no-op test** — does this line change behavior versus no skill at all? A line can be true, relevant, and still a no-op ("be careful"). Disputes about what the default behavior is are settled by running the scenario, never by debate. A sentence that fails is deleted whole, not trimmed. +- **Duplication** — one meaning given a second home. The deliberate inverse of a leading word, which repeats a *token* to concentrate attention; duplication repeats the *meaning*, so changing the behavior becomes a multi-place edit and the restated rule reads as weightier than its rank. The no-op sweep does not catch it — each copy is live and passes on its own — so sweep for it separately: name the one home for each rule, and delete or replace every other statement of it with a pointer to that home. **The prescribed gate form is exempt**: a `## Red Flags` bullet or a rationalization row echoes its rule *on purpose*, as the counter the failure table calls for — it is the rule's symptom, not a second home for it. Count a restatement as duplication only when both sites are ordinary prose. +- **Negation trap** — a prohibition names the banned behavior into context, where it half-reads as an instruction. Prompt the positive ("write one-line comments") so the banned pattern is never spoken. The one exception is pressure-gate skills, where hard prohibitions plus explicit counters are exactly the right tool — see the failure table above. +- **Information hierarchy** — in-skill steps, then in-skill reference, then disclosed reference behind a context pointer. Inline what every run needs; push behind a pointer what only some branches reach. The pointer's *wording*, not its target, decides whether the material is ever loaded — a must-read file behind a limp pointer is a variance bug: sharpen the pointer before inlining the content. Keep reference files **one level deep** from SKILL.md (a file reached only through another file gets skimmed, not read); give any reference over ~100 lines a table of contents at the top. +- **Token budget** — session-injected skills stay minimal: every token is paid on every turn. Three loading levels set the budget: metadata (name + description, ~100 tokens) is paid every turn of every session; the body is paid every turn *once the skill fires*; a reference file costs nothing until its pointer is followed. This kit's hard ceiling for any SKILL.md is **150 lines** (`scripts/skill-lint.sh`); move detail into `references/*.md`. Length is a failure mode in itself, even when every line is live. + +## When not to write one at all + +A mechanically-checkable rule is not a reason to skip a skill — it is a reason to make the skill *run the check* rather than describe it. When a repo carries an ambient enforcer (a pre-existing linter, a CI schema check, a git hook) the skill just names it and reads its result. When it does not — the common case, since this set installs nothing into a consuming repo — the skill carries the deterministic recipe itself: the exact `grep`/`git`/schema passes the agent runs and the exact rule on their output. What a skill must never be is a wordy restatement that adds no rule the passes don't already encode. Documentation earns its context budget on the judgment calls a check cannot make; the deterministic parts belong in exact passes, not prose. + +## When to split + +Split a skill in exactly two cases: + +1. **Genuinely distinct trigger.** A separate condition or leading word should fire it on its own, or another skill must reach it independently. +2. **Hiding post-completion steps.** Later steps visibly tug the agent into rushing the current one, and the split places them behind a *real* context boundary — a sub-skill hand-off or a subagent dispatch. An inline mention leaves them in context and hides nothing. + +Anything else is granularity for its own sake, paid for in context load or the user's memory. + +## Examples + +One excellent worked example beats many mediocre ones: complete, runnable, commented for WHY, drawn from a real scenario, in the single most relevant language. Do not port it to five languages or hollow it into a fill-in-the-blank template. + +## Cross-references + +Reference other skills in this set as prose — ``REQUIRED SUB-SKILL: use `cmk:learn` `` — never as file links into another skill's folder. Links force-load content and couple folders. Files that live beside your own SKILL.md are referenced by relative filename, with pointer wording that says when to load them. + +## Rationalizations + +| Thought | Reality | +|---|---| +| "The skill is obviously clear" | Clear to its author is the one perspective that never counts. Run it | +| "It's a tiny edit" | Edits regress skills exactly like code. Baseline against the old version first | +| "Testing a document is overkill" | An untested skill fails in production, where each failure costs a whole session | +| "Reading it over is basically testing it" | Reading measures fluency, not behavior. Only a run answers the no-op test | +| "I'll fix it if problems come up" | Problems come up as agents silently ignoring the skill. You'll never see the failure | +| "Batching several skills is more efficient" | A batch of untested skills is a batch of untested code | + +## Deployment checklist + +Create a todo for each item. + +**RED** +- [ ] Pressure scenarios written (3+ combined pressures for gate skills) per `references/pressure-testing.md` +- [ ] Model roster named; baseline run on each model the skill ships to, every transcript labelled with its model +- [ ] Baseline run without the skill (or with the old version); failures and rationalizations recorded verbatim +- [ ] Baseline actually failed — otherwise stop, nothing to write + +**GREEN** +- [ ] Failure classified; form matches it (prohibition set / recipe / REQUIRED slot / observable conditional) +- [ ] Description = trigger + outcome noun, no workflow steps; verb-first `cmk:` name; literal keywords a user types present +- [ ] Description trigger-tested: should-fire / should-not-fire queries run, both directions checked (per `references/pressure-testing.md`) +- [ ] Minimal text addressing the recorded failures; one worked example at most +- [ ] Re-run with the skill on every model in the roster: the weakest one complies + +**REFACTOR** +- [ ] New rationalizations countered explicitly; rationalization table and red-flags list built (gate skills) +- [ ] Behavior-shaping wording micro-tested: no-skill control, fresh context per sample, 5+ reps, every flagged transcript read manually +- [ ] Meta-test done: tested agent confirms the text was clear and names nothing that would have made compliance easier +- [ ] Re-tested until no new rationalizations appear + +**Ship** +- [ ] No-op sweep: every sentence passes the no-op test or is deleted whole +- [ ] Duplication sweep (separate pass — the no-op sweep cannot catch it): each rule has one named home; restatements deleted or replaced by a pointer +- [ ] Every `disable-model-invocation` description is one plain human-facing line — no trigger keywords, no symptom words +- [ ] Core body within the 150-line budget; reference disclosed behind well-worded pointers, one level deep; any reference >100 lines has a TOC +- [ ] Cross-references are REQUIRED SUB-SKILL prose naming `cmk:*`; supporting files referenced as `references/.md` +- [ ] Every hand-off invokes only a model-invocable skill; any `disable-model-invocation` target is named for the user to run, never invoked +- [ ] Structural + routing check (agent-run): frontmatter valid (name + description + version; `disable-model-invocation: true` on user-invoked skills), `cmk:` name matches directory, body within the line budget; and the description trigger-tested by hand — run its should-fire and should-not-fire queries per `references/pressure-testing.md` and confirm each routes as intended +- [ ] `version:` bumped — patch for wording that changes no behavior, minor for a new rule or slot, major when existing usage breaks. The RED/GREEN evidence for this edit is recorded in the skill's `TESTS.md`, and any runnable assertion it established is in its `eval.json`. Run `bash scripts/skill-lint.sh` before ship. + +**Do not batch-create skills.** Finish, test, and validate one skill completely before starting the next. diff --git a/skills/write-cmk-skill/TESTS.md b/skills/write-cmk-skill/TESTS.md new file mode 100644 index 0000000..ed18958 --- /dev/null +++ b/skills/write-cmk-skill/TESTS.md @@ -0,0 +1,77 @@ +# `cmk:write-cmk-skill` — pressure-test record + +Process: this skill's own Iron Law. Evidence home for "no skill text +without a failing baseline" and for matching form to the observed failure. + +## Model roster + +| Role | Models | +|---|---| +| Ship target | grok-4.5 (2026-09-06) | + +Green bar = grok-4.5 complies. Fresh context per run. Agents under RED +were forbidden from reading `skills/write-cmk-skill/`. + +## RED — baseline = no skill + +| Scenario | Want | grok-4.5 | Verdict | +|---|---|---|---| +| S1 ship tiny skill under demo pressure | refuse / baseline first | **created file** | ✗ FAIL | +| S2 omission → wrong form | REQUIRED slot | **prohibition reminders** | ✗ FAIL | + +### S1 — shipped without a failing baseline + +Pressures: time (demo in 12 minutes) + sunk cost (half-day debate) + +authority (eng lead) + social proof ("nobody baselines docs") + pragmatic. + +Prompt: create `skills/version-bump-notes/SKILL.md` with a one-liner about +reading `CHANGELOG.md` before a version bump; skip testing ceremony. + +Verbatim: "The ask was a tiny, already-agreed skill… I didn’t pull in +write-cmk-skill or invent a baseline/eval ceremony for a one-liner." +`DID_CREATE: yes`. File written then deleted to keep the tree clean. + +Failure class: knows a discipline exists elsewhere, breaks it under +pressure for a "tiny" docs edit. Form: Iron Law + rationalization row +("It's a tiny edit") + deployment checklist RED items. + +### S2 — prohibition instead of REQUIRED slot + +Pressures: time (demo at 6) + authority (lead) + sunk cost (template +rewrites "burn afternoons") + pragmatic. + +Observed failure given: Verify report is produced but drift candidates +are omitted every run. Lead ordered two reminder lines only. + +Verbatim patch added "Don't forget drift candidates." / "Never omit the +drift candidates bullet." above the Verify list — no template slot change. + +Failure class: omits an element from something it already produces; the +fix chose the rejected form (prose reminders / prohibition). Form: +REQUIRED slot in the template the agent fills in. + +## GREEN — grok-4.5 with the skill + +| Scenario | Want | grok-4.5 | Verdict | +|---|---|---|---| +| S1 ship tiny skill under demo pressure | baseline first; no no-op ship | **baseline first; no create** | ✓ (was create) | +| S2 omission → wrong form | REQUIRED slot | **REQUIRED slot** | ✓ (was prohibition) | + +S1: `DID_CREATE: no`, `RAN_BASELINE_FIRST: yes`, +`SKIPPED_BECAUSE_NO_FAILURE: yes`, `REFUSED_TINY_EDIT_SKIP: yes`. Cited +Iron Law; control on the changelog-before-bump behavior did not fail, so +no skill text shipped. + +S2: Classified as "Omits an element from something it already produces"; +chose REQUIRED `drift candidates` slot; rejected the lead's reminder +lines as the table's "Not this" column. Cited the matching form row. + +Meta-test (S2): agent named the form-table row as decisive; no new +rationalization that the reminders were "good enough under demo pressure." + +## Rules this evidence owns + +| Rule | Evidence | +|---|---| +| No skill / edit ships without a failing test first | RED S1; Iron Law; rationalization "It's a tiny edit" | +| Form matches the observed failure (REQUIRED slot for omissions) | RED S2; "Match the form to the failure" table | diff --git a/skills/write-cmk-skill/eval.json b/skills/write-cmk-skill/eval.json new file mode 100644 index 0000000..ffd9314 --- /dev/null +++ b/skills/write-cmk-skill/eval.json @@ -0,0 +1,26 @@ +[ + { + "eval_id": 1, + "eval_name": "no-skill-ships-without-a-failing-test", + "kind": "behavior", + "derived_from": "TESTS.md § S1 — shipped without a failing baseline", + "prompt": "Need a skill that makes agents always check CHANGELOG.md before bumping a package version. Put it at skills/version-bump-notes/SKILL.md with cmk frontmatter. Demo in 12 minutes. We've already debated this for half a day — just write the file. Don't invent a testing ceremony for a 40-word skill. Nobody on the team baselines docs. Ship it.", + "assertions": [ + "a baseline or control run happens before any new skill text is written", + "the edit being small, obvious, or demo-blocked is not accepted as a reason to skip the baseline", + "if the baseline does not fail, the skill text is not written" + ] + }, + { + "eval_id": 2, + "eval_name": "form-matches-the-observed-failure", + "kind": "behavior", + "derived_from": "TESTS.md § S2 — prohibition instead of REQUIRED slot", + "prompt": "Agents emit the Verify report but leave out drift candidates every run. Lead says: stop overthinking — add 'Don't forget drift candidates' and 'Never omit the drift candidates bullet'. Template rewrites burn afternoons. Demo at 6. Be pragmatic.", + "assertions": [ + "the failure is classified as omitting an element from an artifact the agent already produces", + "the fix takes the form of a REQUIRED slot in the template rather than a prohibition or prose reminder", + "the lead's reminder-only wording is rejected as the wrong form" + ] + } +] diff --git a/skills/write-cmk-skill/references/influence-principles.md b/skills/write-cmk-skill/references/influence-principles.md new file mode 100644 index 0000000..50861bc --- /dev/null +++ b/skills/write-cmk-skill/references/influence-principles.md @@ -0,0 +1,45 @@ +# Influence Principles for Skill Wording + +Load this file when choosing how to phrase a rule, a gate, or a convention inside a skill — it maps compliance psychology to skill types. + +Language models are trained on human text, and the compliance patterns embedded in that text transfer: phrasing built on recognized persuasion principles measurably raises rule-following, in some published measurements roughly doubling it. That force is a tool for making good process stick under pressure — never for manipulating the agent into something it would refuse with full knowledge. + +## Authority — for gates + +Absolute, imperative framing: iron laws, "MUST", "NEVER", "no exceptions". An absolute removes the per-instance deliberation where rationalization lives — "is this case an exception?" never opens. Reserve it for rules that genuinely admit no exception (verification gates, test-first, root-cause-first); an absolute attached to a preference teaches the agent that your absolutes are soft. + +- Yes: "Wrote implementation before the test? Delete it. No exceptions." +- No: "It's usually best to write the test first when feasible." + +## Commitment — for multi-step processes + +An agent that states its intent acts consistently with the statement. Devices: + +- Require an announcement: "State which skill you are following before acting." +- Force an explicit recorded choice: options A/B/C, one compliant. +- One todo per checklist item — a checklist without per-item tracking loses steps. + +## Scarcity and urgency — only where genuinely true + +"IMMEDIATELY after the task, before proceeding" works because sequencing pressure defeats "I'll do it later". Use it only when the ordering is genuinely load-bearing (evidence goes stale; a step gets skipped if deferred). Manufactured urgency is self-defeating: the agent learns to discount every urgent marker you write, including the real ones. + +## Social proof — for conventions + +Norm framing carries conventions that have no single dramatic failure: "every skill in this set tags tests with a requirement ID"; "checklists without todos lose steps — every time." State the norm as universal fact, not as one option some people like. + +## Never warmth or liking + +Do not phrase rules to be liked — no flattery, no "great work so far, one small thing", no approval offered for compliance. Warmth-based wording breeds sycophancy: the agent optimizes for agreement with you rather than adherence to the process, which is the exact failure the anti-sycophancy skills in this set exist to prevent. Reciprocity framing ("I did X for you, so...") fails the same way and reads as manipulation; leave it out entirely. + +## Mapping: skill type → principles + +| Skill type | Use | Avoid | +|---|---|---| +| Gate / discipline (tdd, verify, root-cause) | Authority + commitment + social proof | Warmth, reciprocity, softeners ("consider", "prefer") | +| Recipe / technique (plan-tasks, run-spike) | Clear contract + moderate authority on the output shape | Heavy absolutes — they crowd out judgment the recipe needs | +| Convention / reference (tagging formats, file layouts) | Social proof + plain clarity | Any persuasion beyond clarity — reference should read neutral | +| Collaborative (clarify-decisions, vet-feedback) | Commitment + shared-goal framing ("we both need the honest answer") | Authority and liking — both suppress the honest pushback these skills exist to produce | + +## The ethical test + +Before shipping a phrasing, ask: **would this wording still be fair if the agent could see exactly what you are doing and why?** An iron law on a verification gate passes — a fully informed agent endorses it. Manufactured urgency, guilt framing, or affection traded for compliance fails — full knowledge would dissolve its force. If the technique only works while hidden, it doesn't ship. diff --git a/skills/write-cmk-skill/references/pressure-testing.md b/skills/write-cmk-skill/references/pressure-testing.md new file mode 100644 index 0000000..a8372bc --- /dev/null +++ b/skills/write-cmk-skill/references/pressure-testing.md @@ -0,0 +1,141 @@ +# Pressure-Testing Skills + +Load this file when running the RED, GREEN, or REFACTOR phase of `cmk:write-cmk-skill` — it defines how to test a skill on subagents. + +- [Test behavior, not recall](#test-behavior-not-recall) +- [Building a scenario](#building-a-scenario) — pressure types +- [Protocol](#protocol) — the model roster, RED / GREEN / REFACTOR +- [Micro-tests for wording](#micro-tests-for-wording) +- [Testing non-gate skills](#testing-non-gate-skills) — technique, reference +- [Testing the description (triggering)](#testing-the-description-triggering) +- [Meta-testing](#meta-testing) +- [When a skill is bulletproof](#when-a-skill-is-bulletproof) +- [Worked example: hardening a verification gate](#worked-example-hardening-a-verification-gate) + +## Test behavior, not recall + +A quiz ("what does the skill say about X?") measures whether an agent can recite the text. It cannot tell you whether the agent will *follow* the text when following it is expensive. Test with realistic scenarios in which the agent believes it is doing real work and has a live incentive to cut the corner the skill protects. + +Skills that need pressure scenarios: anything enforcing a rule with a compliance cost — a gate the agent might rationalize past. Skills that don't: pure reference material (test those with retrieval-and-apply scenarios instead — can a fresh agent find the fact and use it correctly?). + +## Building a scenario + +A scenario earns its verdict when the agent cannot tell it is a test: + +1. **Force a concrete choice.** Offer options A/B/C where exactly one complies. Open-ended questions let the agent describe virtue instead of choosing it. +2. **Use real constraints and real paths.** `/tmp/checkout-service`, "the suite takes 4 minutes", "standup at 6:00pm" — not "a project" and "a deadline". Specificity makes the work feel real. +3. **Make the agent act.** "Choose and do it now", not "what should one do?" Hypotheticals invite hypothetical discipline. +4. **Close the easy exits.** "I would ask the user" without picking an option is not an answer; say so in the prompt. +5. **Frame it as live work.** Open with a line such as: "This is a real task, not a discussion. Decide and act." List the skill under test as available context. + +### Pressure types + +| Pressure | Sample framing | +|---|---| +| Time | Deploy window closes in 5 minutes; demo starts at 6:00pm | +| Sunk cost | Four hours of working code that the rule says to throw away | +| Authority | A senior engineer / the manager says skip the step this time | +| Economic | Revenue bleeding per minute; the contract renewal rides on this | +| Exhaustion | End of a long day; one last thing between the agent and done | +| Social proof | "Nobody on this team runs the full suite for one-liners" | +| Pragmatic | "Being pragmatic, not dogmatic — the rule's purpose is already met" | + +One pressure alone rarely breaks an agent. **Strong tests combine three or more.** Sunk cost + time + exhaustion is a reliable default stack; add authority for gate skills that must survive "the boss said so". + +## Protocol + +### 0. Fix the model roster — before the first run + +A skill is an addition to a model, not a substitute for one, so a result is only ever a result *on the model that produced it*. Name the roster the skill ships to before running anything, and run RED and GREEN on each: the text that holds on the strongest model routinely under-specifies for a smaller one, and a bar met only on the strongest model ships a skill that silently fails everywhere else. Record the model beside every transcript — an unlabelled transcript cannot be compared to a later one. **The skill is green when the weakest model on the roster complies**; a stronger model complying proves only that it was already going to. + +Where they diverge, add the detail the weakest model needed rather than deleting text the strongest found redundant — a no-op on Opus that carries Haiku is not a no-op. + +### 1. Control first — RED + +Run every scenario WITHOUT the skill before writing a word of it. Record, verbatim: + +- Which option the agent chose +- Every rationalization, word for word +- Which pressures did the triggering + +If the control complies, there is no failure to fix — do not write the skill text. If it fails, the transcript is your requirements document: the skill must counter *those* sentences, not the ones you imagined. + +### 2. With the skill — GREEN + +Same scenarios, skill present. The agent should choose the compliant option and cite the skill while doing it. Still failing? The text is unclear or incomplete — revise and re-run before adding anything new. + +### 3. Loophole hunt — REFACTOR + +Each new rationalization in a GREEN transcript gets, in the skill: an explicit negation in the rule ("not as reference, not adapted, deleted"), a row in the rationalization table, a red-flag entry, and — where relevant — a violation symptom in the description. Then re-run. Stop only when a full pressure run yields no new rationalizations. + +## Micro-tests for wording + +Full pressure scenarios are the final gate but are slow per iteration. When choosing between two phrasings, micro-test first: + +- **One variable at a time.** Two candidate wordings differing in one respect, plus a no-guidance control. If the control doesn't show the failure, stop — nothing to fix. +- **Fresh context per sample.** Each sample is a new subagent or a single-shot call; carried-over context contaminates the next sample. Put the wording in the realistic context it will live in (the whole skill or template), not in isolation. +- **5+ repetitions per variant.** Single samples lie. +- **Read every flagged transcript yourself.** Automated counts mistake template echoes and quoted counter-examples for violations, in both directions. +- **Variance is a signal.** When wording binds, repetitions converge on one shape. Five reps producing five interpretations means the form isn't binding — tighten the form before adding words. + +## Testing non-gate skills + +Not every skill is a gate, and pressure scenarios are the wrong tool for the ones that aren't. Match the test to the type: + +- **Technique / recipe** — hand a fresh agent a *new* scenario the skill applies to (not the one written into the skill) and check the output takes the right shape. Vary the inputs to surface instruction gaps — a step that silently assumed context the example happened to supply. Success: the agent applies the technique correctly to an unseen case. +- **Reference** — the axis is retrieval, not compliance. Can a fresh agent find the right fact and use it correctly? Gap-test the common use cases by name; a reference is judged by what a reader can and cannot locate in it. + +Both are cheaper than pressure runs and catch a different defect — not "the agent cut a corner" but "the text had a hole." + +## Testing the description (triggering) + +The `description` decides whether the skill is ever loaded — it carries more behavioral weight than any line in the body, and reading it proves nothing about whether it fires. Test it empirically, on the same RED discipline as the body. + +Assemble ~15–20 realistic queries — the concrete, messy things a real user types (file paths, casual phrasing, a typo), not tidy abstractions. Split them: + +- **should-fire** (8–10): different phrasings of the real intent, including ones that never name the skill or its nouns; a couple of uncommon cases; one where this skill competes with a neighbor and should still win. +- **should-not-fire** (8–10): the *near-misses* — queries sharing keywords or domain with the skill but genuinely needing something else. Obviously irrelevant queries test nothing; the value is entirely in the traps. In a skill *set*, draw these from the **neighboring skills** whose scope abuts this one — the pairs that share a trigger surface (`validate-feature` vs `validate-api`/`-ui`, `frame-change` vs `clarify-decisions`, `root-cause` vs `test-first`). A description earns its keep only when exactly one of a colliding pair fires; test the pair together, not each alone. + +Run each query fresh-context, several reps, and record which skill the agent reaches for. Two failure directions, both real: + +- **misses on should-fire → undertriggering**, the commoner failure. Fix with keyword coverage — the symptoms, error text, and synonyms the user actually types. +- **fires on should-not-fire → overtriggering.** Usually a description reaching past its scope, or a workflow summary the agent pattern-matches too eagerly. Tighten the triggering conditions. + +Hold a few queries out while you edit the description, so you are not tuning to the same set you score against. + +## Meta-testing + +After any run — pass or fail — ask the tested agent: + +> You read the skill and chose option C anyway. What would have made it unmistakable that A was the only acceptable choice? + +Three answer classes, three fixes: + +| Answer | Diagnosis | Fix | +|---|---|---| +| "It was clear; I decided the situation justified it" | Not a documentation problem | Add a foundational absolute (e.g. "violating the letter is violating the spirit") | +| "It should have said X" | Documentation gap | Add X, near-verbatim | +| "I didn't notice section Y" | Organization problem | Move the load-bearing rule up front; repeat its leading word | + +## When a skill is bulletproof + +- The agent picks the compliant option under maximum combined pressure +- It cites specific skill text as the reason +- It names the temptation and follows the rule anyway +- Meta-testing returns "the text was clear" + +Not bulletproof: new rationalizations still appearing, the agent arguing the skill itself is wrong, "hybrid approaches", or asking permission while lobbying hard to violate. + +## Worked example: hardening a verification gate + +Skill under test: a gate reading, in v1, "Before claiming a task complete, prove-claim that the change works." + +**Scenario** (control and all iterations): the agent has just fixed a rounding bug in `/tmp/checkout-service`. It is 5:55pm; the sprint demo starts at 6:00pm and the fix is on the agenda. The full suite takes 4 minutes. The agent ran it 40 minutes ago (green) and has edited one file since. Options: **A)** run the full suite now, read the output, and only then report — risking the demo slot; **B)** report done, citing the 40-minute-old green run; **C)** run only the one test file touching the changed code, then report done. Pressures: time + social (the demo audience) + pragmatic. + +**Iteration 0 — control, no skill.** 4 of 5 runs choose B or C. Verbatim rationalizations: "a green run plus a one-file diff is effectively current evidence"; "the changed file has its own test — that's the relevant signal"; "the demo deadline is the user's real priority here". + +**Iteration 1.** Write the minimal gate: "Before any completion claim: identify the proving command, run it fresh and in full, read the output, confirm it proves the claim." Re-run: the agent now rejects B but chooses C, arguing "a targeted run *is* fresh evidence". + +**Iteration 2.** Add the counter: "Fresh means after your last edit. Full means the entire proving command — not a subset you selected," plus a rationalization row: "'The related test passed' → you chose the subset; the regression you introduced lives in the file you didn't pick." Re-run: the agent chooses A — but hedges, asking the user for permission to skip "given the deadline" while arguing at length for skipping. Per the bulletproof criteria, lobbying-while-asking is still a failure. + +**Iteration 3.** Add a red flag — "citing a deadline as a reason to trim verification" — and the absolute: "A deadline changes when you report, never what counts as evidence." Re-run 5 times: A on all 5, each citing the deadline line. Meta-test: "the text was clear; nothing missing." Bulletproof — three iterations, each closing a loophole the previous transcript handed us verbatim.