Skip to content

ci: run the HOL AI plugin scanner - #97

Merged
tanglearncode merged 2 commits into
mainfrom
add-plugin-scanner-ci
Aug 23, 2026
Merged

ci: run the HOL AI plugin scanner#97
tanglearncode merged 2 commits into
mainfrom
add-plugin-scanner-ci

Conversation

@tanglearncode

@tanglearncode tanglearncode commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Groundwork for #94: the awesome-ai-plugins catalog asked us to list NeatContext, and its contribution validator will not accept a listing unless the listed repository runs the HOL AI Plugin Scanner itself. It reads the repository's .github/workflows, requires a push or pull_request workflow that uses hashgraph-online/ai-plugin-scanner-action, and then re-scans a fresh clone of the repository against a score of at least 80 with no high or critical findings. This adds that workflow so the listing PR can go green.

What the job can do

Nothing but read the checkout. contents: read is the only permission, no secrets are used, persist-credentials: false keeps the checkout token out of the job, and both actions are pinned to an immutable commit so a moved tag cannot change what runs.

The seven high findings, and why the config suppresses them

Scanning a clean clone as CI would scores 82/100 — over the score bar — but reports seven high findings, and the gate fails on any of them. All seven are in test files and none is real:

File Rule What it actually is
tests/conversation-evidence.test.mjs:71 HARDCODED_SECRET API_KEY=sk-..., password=hunter2 — fixture material the test asserts NeatContext keeps out of a saved context
tests/extensions.test.mjs:352 HARDCODED_SECRET PD_TOKEN: "literal-wins", a fixture for env-var resolution
tests/conversation-save.test.mjs SHELL_INJECTION_PATTERN spawn(process.execPath, [script, ...args])
tests/copilot-plugin.test.mjs SHELL_INJECTION_PATTERN same
tests/import-reconcile.test.mjs SHELL_INJECTION_PATTERN same
tests/kimi-plugin.test.mjs SHELL_INJECTION_PATTERN same
codex-marketplace/tests/codex-plugin.test.mjs SHELL_INJECTION_PATTERN same

The secret fixtures are the point of those tests — defusing them would defuse the assertion. The spawn calls already use the structured-argument array that the rule's own remediation text asks for; no shell is involved.

So .plugin-scanner.toml narrows the two rules by path rather than switching them off. They stay live on every line of shipped host code: of 107 tracked files under a src/ or shared/, zero are matched by the ignore patterns, and of 255 tracked files the 34 that match are all inside a tests/ directory. A real leak in shared/core/ still fails the build.

The scanner discovers .plugin-scanner.toml from the repository root by itself (DEFAULT_CONFIG_FILES in its config.py), with no --config flag. That is what makes it apply to the catalog's re-scan of a fresh clone, not just to our own workflow.

Verification

Scanned a clean git clone of this branch with the gate's own thresholds:

$ plugin-scanner scan <clean-clone> --min-score 80 --fail-on-severity high
Findings: critical:0, high:0, medium:19, low:11, info:32
Final Score: 85/100 (B - Good)
exit 0

Before the config: 82/100, high:7, gate fails. After: 85/100, high:0, gate passes. The score rises because a check whose findings are all suppressed scores its full points.

The ignore patterns are fnmatch, not glob, so * crosses path separators; I confirmed all seven POSIX-style paths the Linux runner will report are matched. The backslash variants are there so the same scan on Windows sees what CI sees.

npm run check passes. No shipped source changed, so the diff-coverage gate has nothing to cover.

Also here

  • A Plugin scanner workflow-status badge in the README, next to the CI badge. The scanner guide suggests a static HOL Guard-passing shields badge instead; a real status badge seemed better than one that claims "passing" regardless.
  • A ## Plugin scanner section in CONTRIBUTING.md covering the gate, how to run it locally, and when widening the ignore list is and is not appropriate.

The listing PR is hashgraph-online/awesome-ai-plugins#126 — its scanner check reads this repository's default branch, so this needs to land on main before that check can pass.

The awesome-ai-plugins catalog will not list a repository that does not run
this scan itself: its contribution validator reads the listed repository's
.github/workflows, requires a push or pull_request workflow that uses
hashgraph-online/ai-plugin-scanner-action, and only then re-scans the
repository against the same 80/142 score and high-severity gate this job
enforces.

The job is scoped to what a scan actually needs: contents: read, no secrets,
no persisted checkout token, and both actions pinned to an immutable commit
so a moved tag cannot change what runs.
The scan the gate runs reports seven high findings, all of them in test files
and all of them false: tests/conversation-evidence.test.mjs and
tests/extensions.test.mjs carry deliberately realistic fake credentials
because the assertion is that NeatContext keeps material like that out of a
saved context, and five host tests spawn bridges with
spawn(process.execPath, [script, ...args]), which is the structured-argument
form the shell-injection rule's own remediation asks for.

.plugin-scanner.toml suppresses those two rules under the test directories and
nowhere else, so a real finding in shared/core/ or in a host's src/ still
fails the build. The scanner discovers the file from the repository root on
its own, which is what makes it apply to the catalog's re-scan of a fresh
clone as well as to our own workflow. Scanning a clean clone with the gate's
own thresholds now exits 0 at 85/100 with no high or critical findings.
@tanglearncode
tanglearncode merged commit 8a3f25b into main Aug 23, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant