ci: run the HOL AI plugin scanner - #97
Merged
Merged
Conversation
The awesome-ai-plugins catalog will not list a repository that does not run this scan itself: its contribution validator reads the listed repository's .github/workflows, requires a push or pull_request workflow that uses hashgraph-online/ai-plugin-scanner-action, and only then re-scans the repository against the same 80/142 score and high-severity gate this job enforces. The job is scoped to what a scan actually needs: contents: read, no secrets, no persisted checkout token, and both actions pinned to an immutable commit so a moved tag cannot change what runs.
The scan the gate runs reports seven high findings, all of them in test files and all of them false: tests/conversation-evidence.test.mjs and tests/extensions.test.mjs carry deliberately realistic fake credentials because the assertion is that NeatContext keeps material like that out of a saved context, and five host tests spawn bridges with spawn(process.execPath, [script, ...args]), which is the structured-argument form the shell-injection rule's own remediation asks for. .plugin-scanner.toml suppresses those two rules under the test directories and nowhere else, so a real finding in shared/core/ or in a host's src/ still fails the build. The scanner discovers the file from the repository root on its own, which is what makes it apply to the catalog's re-scan of a fresh clone as well as to our own workflow. Scanning a clean clone with the gate's own thresholds now exits 0 at 85/100 with no high or critical findings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groundwork for #94: the awesome-ai-plugins catalog asked us to list NeatContext, and its contribution validator will not accept a listing unless the listed repository runs the HOL AI Plugin Scanner itself. It reads the repository's
.github/workflows, requires apushorpull_requestworkflow that useshashgraph-online/ai-plugin-scanner-action, and then re-scans a fresh clone of the repository against a score of at least 80 with no high or critical findings. This adds that workflow so the listing PR can go green.What the job can do
Nothing but read the checkout.
contents: readis the only permission, no secrets are used,persist-credentials: falsekeeps the checkout token out of the job, and both actions are pinned to an immutable commit so a moved tag cannot change what runs.The seven high findings, and why the config suppresses them
Scanning a clean clone as CI would scores 82/100 — over the score bar — but reports seven high findings, and the gate fails on any of them. All seven are in test files and none is real:
tests/conversation-evidence.test.mjs:71HARDCODED_SECRETAPI_KEY=sk-...,password=hunter2— fixture material the test asserts NeatContext keeps out of a saved contexttests/extensions.test.mjs:352HARDCODED_SECRETPD_TOKEN: "literal-wins", a fixture for env-var resolutiontests/conversation-save.test.mjsSHELL_INJECTION_PATTERNspawn(process.execPath, [script, ...args])tests/copilot-plugin.test.mjsSHELL_INJECTION_PATTERNtests/import-reconcile.test.mjsSHELL_INJECTION_PATTERNtests/kimi-plugin.test.mjsSHELL_INJECTION_PATTERNcodex-marketplace/tests/codex-plugin.test.mjsSHELL_INJECTION_PATTERNThe secret fixtures are the point of those tests — defusing them would defuse the assertion. The
spawncalls already use the structured-argument array that the rule's own remediation text asks for; no shell is involved.So
.plugin-scanner.tomlnarrows the two rules by path rather than switching them off. They stay live on every line of shipped host code: of 107 tracked files under asrc/orshared/, zero are matched by the ignore patterns, and of 255 tracked files the 34 that match are all inside atests/directory. A real leak inshared/core/still fails the build.The scanner discovers
.plugin-scanner.tomlfrom the repository root by itself (DEFAULT_CONFIG_FILESin itsconfig.py), with no--configflag. That is what makes it apply to the catalog's re-scan of a fresh clone, not just to our own workflow.Verification
Scanned a clean
git cloneof this branch with the gate's own thresholds:Before the config: 82/100,
high:7, gate fails. After: 85/100,high:0, gate passes. The score rises because a check whose findings are all suppressed scores its full points.The ignore patterns are
fnmatch, not glob, so*crosses path separators; I confirmed all seven POSIX-style paths the Linux runner will report are matched. The backslash variants are there so the same scan on Windows sees what CI sees.npm run checkpasses. No shipped source changed, so the diff-coverage gate has nothing to cover.Also here
Plugin scannerworkflow-status badge in the README, next to the CI badge. The scanner guide suggests a staticHOL Guard-passingshields badge instead; a real status badge seemed better than one that claims "passing" regardless.## Plugin scannersection inCONTRIBUTING.mdcovering the gate, how to run it locally, and when widening the ignore list is and is not appropriate.The listing PR is hashgraph-online/awesome-ai-plugins#126 — its scanner check reads this repository's default branch, so this needs to land on
mainbefore that check can pass.