feat(docs): replay the Evidence tutorials with the page runner - #1602
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d12b850fa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
registry-stack/docs/site/scripts/run-tutorial.mjs
Lines 184 to 187 in 5cc2a27
Preserve errexit while checking an expected refusal
For a multi-command test-exit fence, disabling errexit for the entire brace group lets execution continue after an unexpected early failure. If a later command happens to return the annotated status, the runner accepts the fence and may execute side effects a reader running under the journey's normal set -e would never reach. Run the fence in a set -e subshell and capture that subshell's status instead.
AGENTS.md reference: docs/site/AGENTS.md:L54-L56
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
test-file writes a block the page shows as a whole file, test-background keeps a fence running beside the journey until its ready URL answers, and test-cwd names the directory the page tells the reader to return to. Every process a journey starts is stopped when it ends, passing or failing. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…mock to tutorial journeys Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…kout scripts Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…tire the bash gate Every replayed Evidence page now carries its own test-file, test-expect, and test-excerpt annotations, the output blocks show what the commands print today, and CI runs run-tutorial.mjs --gate evidence. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…wn live command Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
…into its gate Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
fb9ce1d to
650f5d1
Compare
…is stopped A background fence promises to keep running until the next one starts or the page ends, but stop() only sent the group a signal and reported success even when the group had already ended on its own. A service that crashed after answering its readiness check was silently accepted. Check liveness before stopping and fail with what the command printed when it had already exited. Signed-off-by: Jeremi Joslin <jeremi@joslin.fr>
1 participant
Stacked on #1573 (Casework): the first three commits belong to that PR; review the last four here.
Replaces
check-evidence-tutorials.shwith the page-as-spec runner from #1572. Each replayed Evidence page now declares its journey intutorial_testfrontmatter and checks itself with annotations on the blocks a reader already sees.What changes
test-file(a whole file the reader creates in their editor),test-background="<url>"(a command left running in another terminal; starting another one stops it, and the page end stops them all), andtest-cwd="<dir>"(the directory the page tells the reader to return to).evidence,evidencectl,evidence-oid4vci, and interop-test binaries, the Python client wheel fromREGISTRY_CLIENT_PY_WHEEL, and the FHIR mock on 8003 for every journey.test-file,test-expect, andtest-excerpt; the old SPEC_ASSERTS became checks on output blocks the page shows. Several blocks had drifted from what the commands print today and now show the real output:audience,concept,handle,required,responseFormats);TASK GRANT REFUSED);The 12 pages the bash gate never replayed say why in
tutorial_test.skip.check-evidence-tutorials.shand its test are deleted.evidence-tutorial-fence.shstays, because the Relay gate still uses it until Relay migrates.evidence-tutorialsjob runsnpm ci, the runner tests, the dry run, andrun-tutorial.mjs --gate evidence.ci_changes.pyroutes on the runner and the replayed pages, and its test reads the frontmatter instead of parsing the bash registry.Verification
ci_changestests: 118 pass. Gates inventory tests: 70 pass, and the inventory check passes.npm test,check:draft-links,check:markdown,check:content,check:style, and the Evidence anchors check all pass.refuse-unsafe-evidence-requestspasses on retry.control-who-can-request-evidencecould not get past a local ThunderID flake (below), so this job's CI run is the first full replay of that page's fixed blocks.Known local flake (not in this PR)
On macOS with OrbStack,
evidencectl dev startsometimes fails when it restarts in a project directory whose issuer state was just removed. ThunderID'ssetup.shreportsfailed to ping database runtime_transient: unable to open database file (14)on the bind-mounted SQLite file. A 20-second pause does not reliably avoid it.registry-thunderid-toolingdrops setup's stdout because it carries the admin credential, so the supervisor log only says setup did not complete. I have not seen it on Linux.