Skip to content

Keep documentation tooling tests focused on supported behavior - #240

Open
Marius Storhaug (MariusStorhaug) wants to merge 6 commits into
mainfrom
functional-test-alignment
Open

Marius Storhaug (MariusStorhaug) wants to merge 6 commits into
mainfrom
functional-test-alignment

Conversation

@MariusStorhaug

@MariusStorhaug Marius Storhaug (MariusStorhaug) commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Documentation tooling tests exercise real commands and retain package-quality gates without fixed skill inventories or transport call counts. The link-checker tests no longer extract an internal function from source.

The command-level tests found a real underscore-anchor defect. The user approved the narrow heading-helper fix, so valid GitHub anchors keep their literal underscores. No consumer action is required.

Fixed: GitHub heading links keep literal underscores

The cross-repository link checker accepts the correct #snake_case-and-kebab-case anchor and rejects the form that removes the underscore. It preserves literal underscores, code spans, and valid adjacent or nested formatting. GitHub-literal {#custom} text also remains part of the heading instead of becoming a site-specific attribute. Existing links and command inputs need no change.

Technical details

Review-ready: implementation, source-bound audit, local validation, and exact-head CI are complete. All 5 findings from the first two full reviews have fixes, regression cases, replies, and resolved threads. The final exact-head review is APPROVED and states that there are no active/open issues. No merge was performed; GitHub auto-merge and app Agent merge remain off.

Source and scope

Field Verified identity
Initial live main and clean worktree 33ccfaae6d905150f104dd763e34bf36cb2de1f9
Refreshed default-branch baseline 1b0d4893c8fefabe604d04393068a8ff6f848082
Reviewed candidate ee3c17855cb282cdf236fc764df3f45502f4c057
Unchanged Testing policy blob fd153feef3ddac71c587eb2c547623a01379379f

Both complete recursive tree responses report truncated=false and the same six suite files. #237 merged into main during the audit and added two fixed catalog assertions. This branch was refreshed from that merged default-branch commit, not an open PR branch. Its writing, skill, navigation, and manifest changes are baseline context, not this PR's delta.

The scoped diff changes exactly four files: .github/scripts/Test-CrossRepositoryLink.ps1, tests/PluginMarketplace.Tests.ps1, tests/Test-CrossRepositoryLink.Tests.ps1, and tests/Update-GitHubActionPin.Tests.ps1. The user approved only the demonstrated heading-helper defect as a runtime exception. No policy, workflow, pin, security setting, or unrelated production code changed. The draft opened immediately after the first focused commit; the authorized fix, catalog cleanup, and review corrections have separate focused commits.

The helper reuses the Markdig.Signed assembly already bundled with PowerShell. It selects CommonMark plus GitHub's autolink and strikethrough inline extensions, not PowerShell's advanced/site-specific extensions. It does not enable generic attributes or use renderer-generated anchor IDs. The existing GitHub slug algorithm stays unchanged. No parser package, module, runtime, or framework was installed or added.

Compatibility evidence is source-bound: PowerShell v7.0.0 ships Markdig.Signed 0.18.1, and the selected public APIs exist at renderer source dd6418d108c6cb66f2196ad04fad5d59a231e72e. The script's PowerShell 7.0 minimum is unchanged. Current Windows and Linux execution is verified below; no separate PowerShell 7.0 execution is claimed.

Evidence inventory

Suite or group Disposition, real entry point, observed result, and policy reason
Marketplace uniqueness and source resolution Retain 2 existing static package-quality cases through Pester, the JSON reader, and filesystem checks. The current package and an anonymous positive fixture pass; duplicate identities fail. No specific plugin name or count is required.
Manifest identity/version/schema selector Retain the existing public package-profile check. The canonical selector is required by Agent Plugins; matching advertised identity/version follows the marketplace contract. This is static quality, not full-schema or installer coverage.
Agent Skills metadata Retain the required-file and metadata checks, and rename the group to state its limited scope. All 4 package-quality cases pass; no instruction phrase is checked.
Writing skill inventory Remove the count-of-one and fixed writing-skill directory assertions from the newly merged baseline. They do not invoke client discovery or an owned runtime.
Issue/PR/workflow skill inventory Remove the three hard-coded directory requirements and their unused enumeration setup. Do not replace them with another list, source search, snapshot, or validator.
Requires-Modules ranges, exact versions, GUIDs, and exits Retain all 11 real child-PowerShell cases: 3 range, 2 exact-version, 3 identity, and 3 process-exit checks. All pass.
Cross-link broken and resolving targets Retain 4 broken-target and 5 resolving-target command cases, including checkout resolution, branch/README anchors, and cache reuse. All pass against temporary repositories and a loopback service.
GitHub/site slug distinctions Retain all 3 real-command punctuation, wrong-site-slug, and duplicate-heading cases. All pass.
Internal slug cases Rewrite all 14 AST-extracted cases through the complete checker command, with the original expected anchors unchanged. Add 17 positive underscore/formatting/inline-code cases and 2 incorrect-anchor cases. All 33 pass; no source extraction remains.
Cross-link zero-work/scope and service/quota outcomes Retain all 3 scope/empty-work and 4 error/quota command cases. All pass. Request-count checks here measure documented cache/quota behavior, not an arbitrary call sequence.
Documentation link checker Retain all 6 real-command footnote, reference, and inline-link cases on controlled temporary pages. All pass.
Documentation index generator Retain 2 unchanged-byte, 2 stale-input rejection, and 2 LF/CRLF repair cases through the real generator and supported -Check mode. All pass. This does not enforce checked-in index drift.
Action pin updater Retain all 4 real-updater update/encoding/WhatIf/unchanged/error cases on temporary YAML and controlled release responses. All pass. Remove only the exact count of 6 REST calls.
Runner and CI Preserve Pester discovery and zero-work guards, named Test job, lint, local and cross-repository links, index generation, site build, pins, and security settings. Previously removed documentation-content suites and checked-in index-drift enforcement stay removed.

The repository owns no plugin installer or separate production manifest validator. The retained package cases are format-quality safeguards, not proof of client behavior. The required canonical selector follows the Agent Plugins manifest contract; a parsed file plus a fixed string is not presented as behavioral coverage. The package profile, marketplace contract, and Agent Skills specification support these limited checks. The Task holds the complete group-by-group inventory and source evidence.

Validation and full-review disposition

Command or control Actual result at the reviewed candidate
pwsh -NoLogo -NoProfile -File .\.github\scripts\Invoke-PesterSuite.ps1 83 passed, 0 failed, 0 skipped across 6 suites. PowerShell 7.6.9 on Windows; unchanged runner pin Pester 6.0.1.
Focused suite paths through the same runner Pin updater: 4 pass before/after. Marketplace: 6 pass before catalog cleanup, 4 after. Cross-link checker: 52 pass.
Independent anchor oracle Actual github-slugger 2.0.0 at 3461c4350868329c8530904d170358bca1d31448 verifies all 31 positive anchor expectations. Existing Node v26.12.1; no package installation or added dependency.
Independent heading rendering GitHub Markdown API mode=gfm confirms the new headings' visible text, including the review examples and literal generic attributes. This response is not claimed as the automatic-slug oracle. Production renderer output is not used to compute expected anchors.
Anonymous package-quality controls Positive retained-field fixture: exit 0, 4 pass. Duplicate identity: exit 1, 1 expected failure. Empty catalog: exit 1 through both the runner and direct Pester.
Repository static and link checks 0 configured PSScriptAnalyzer diagnostics in all 4 changed files; git diff --check passes. All documentation links resolve across 130 files.
Docs CI Successful pull-request run at ee3c17855cb282cdf236fc764df3f45502f4c057: Lint, Build, Links, and Test succeed. The CI runner reports 83 passed, 0 failed, 0 skipped. Publish is intentionally skipped.
Cross-repository Links CI Same exact head: 7 in-scope links checked across 134 Markdown files; every link resolves.

All 64 original tests passed, but the complete command rejected the valid underscore anchor. Test-first evidence remains in the Task and #241. Correct expected values were retained. An attempted name filter selected 0 cases and is excluded from evidence; counted runner results supply the actual red and green runs.

Full review read Exact head Summary and disposition
First review 9e19827aac3ca91ec3d44680c289324d6e196790 Changes recommended: 4 valid findings for adjacent formatting, nested underscore delimiters, unequal backtick runs, and unmatched backticks. All 7 examples first fail through the command, then pass after renderer reuse. All 4 threads have specific replies and are resolved.
Second review 617914a03a6318e3997800527903480befa5d828 Changes recommended: confirms the 4 earlier fixes; finds 1 valid generic-attribute regression. Correct-anchor acceptance and wrong-anchor rejection first fail (50 pass/2 fail), then pass with GitHub inline syntax (52 pass/0 fail). Its thread has a specific reply and is resolved.
Final review ee3c17855cb282cdf236fc764df3f45502f4c057 APPROVED. The complete summary states No active review issues and No open issues, and confirms the remaining issue is resolved. No new inline comments; all 5 threads are resolved, with complete thread pagination verified.

The clean result is based on the full summary, exact source identity, and resolved threads, not only the runner exit or new-comment count. The native closing references were verified for the Task and its convergent user-approved Bug.

Simulated release responses make mutable latest-release data and invalid/unavailable responses deterministic without live quota use. The loopback service permits safe missing-target, quota-exhaustion, and HTTP-failure cases. These checks prove caller behavior, not live authentication or every vendor failure mode. The existing public read-only CI cross-link job is separate live evidence: Ubuntu 24.04, repository-scoped workflow token, public repository reads, no remote writes or resources to clean up. No live pin-updater API check or plugin-client installation/activation test was run. Metadata checks are not complete schema validation or trigger-reliability evidence. Scratch experiment files were removed after verification; durable evidence stays in GitHub.

Standards and convergence

Changed surface Standards checked Framework or domain docs checked Result
tests/PluginMarketplace.Tests.ps1 Testing; PowerShell Testing; Code Layout; Natural Language Package profile, marketplace contract, Agent Plugins and Agent Skills formats Fixed in this PR
tests/Update-GitHubActionPin.Tests.ps1 Testing; PowerShell Testing; Code Layout Existing updater parameter and output contract Fixed in this PR
tests/Test-CrossRepositoryLink.Tests.ps1 Testing; PowerShell Testing; Naming; Error Handling Checker command help and independent GitHub heading/anchor evidence Fixed in this PR
.github/scripts/Test-CrossRepositoryLink.ps1 heading helper Functions and reuse-before-build; PowerShell; Documentation; Natural Language Existing heading contract, bundled renderer API provenance, GitHub Markdown, pinned slugger Fixed in this PR

The alignment pass re-read the process, testing, PowerShell, and applicable format guidance. .github/CONTRIBUTING.md and the generator's optional -Check guidance already describe the retained commands correctly. Only help tied to the changed heading helper needed an update. No policy prose or repository notes were added. The route builds and deploys Pages; no Release Management block or labels apply.

The scoped open-issue sweep covers tests, tooling, plugins, and anchors. It finds the closing Task and the fully delivered user-approved underscore defect, with no other convergent issue. The unrelated policy PBI #105 and published-URL, AGENTS-routing, navigation, and writing-campaign work remain outside this diff. All implementation, validation, and review plan steps are complete. No scoped blocker remains.

Campaign qualification supplied by the coordinator, not this PR's own audit: complete recursive trees (truncated=false) for MSXOrg/.github main at 0c67251812ea306543f3a14ab473ae357cfb76ac and PSModule/.github main at 4e0092de8f208b11e0d4607ced93471607aa748e contain no test code, runners, scripts, dependency manifests, hooks, or validation workflows. No empty PR is warranted for those snapshots. Open branches, including PSModule/.github#89, were not imported. PSModule/Process-PSModule has a separate scoped audit; this PR covers only MSXOrg/docs.

Related references

This was referenced Nov 4, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Markdown delimiter parsing mishandles adjacent formatting, nested emphasis, and backticks, causing the link checker to generate incorrect anchors and reject valid links.

4 open issues
What changed in this PR

This PR shifts documentation-tooling tests toward supported command behavior while retaining package-quality checks. It also updates heading normalization to preserve literal underscores in GitHub anchors.

Changes:

  • Replace internal slug-function extraction with command-level anchor tests, including underscore and inline-code cases.
  • Remove fixed skill-inventory assertions and the pin updater’s exact REST-call count.
  • Refine underscore-emphasis handling in the cross-repository link checker.
File Description
tests/​Update-GitHubActionPin.Tests.ps1 Removes the exact REST-call-count assertion.
tests/​Test-CrossRepositoryLink.Tests.ps1 Moves slug coverage to checker commands and adds anchor regressions.
tests/​PluginMarketplace.Tests.ps1 Removes fixed skill inventories and clarifies metadata-check scope.
.github/​scripts/​Test-CrossRepositoryLink.ps1 Preserves literal underscores while handling emphasis and code spans.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The link checker strips heading attributes that GitHub preserves as literal text, causing it to reject valid anchors and accept nonexistent ones.

1 open issue
4 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approved

No active review issues are recorded for the documentation-tooling tests and heading-anchor changes.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Preserve in-word underscores when checking GitHub heading anchors [functional-test-alignment] Test supported documentation tooling behavior

2 participants