Four scanners could not say they had not looked, and guarddog could not say it had found something

[HackingGate]

tool() mapped every non-zero exit to a verdict, which files "I could not run" under "you are out of step". Checking that assumption meant running all five scanners rather than reading their docs, and four of them break it. One breaks it the other way.

guarddog was reporting findings as clean

guarddog verify exits 0 whether it found three high-severity risks or none, in both ecosystems. Verified: pyyaml==6.0.2 gives three high risks at score 8.8 / high_risk, and exits 0. npm node-gyp gives 2 risks, exit 0.

This section answered by exit code, so every guarddog finding in the fleet was reported clean. That is a false negative on the one scanner here whose subject is malware and typosquats.

--exit-non-zero-on-finding is not the remedy. It counts issues, which includes capability matches. six==1.17.0 reports issues: 2 with risks: [] and guarddog's own label no_risks_detected, so the flag fails a package guarddog itself calls clean. It trades a false negative for a false positive, and there is a test pinning that.

guarddog now runs with --output-format json and its own risks list is counted. This is the one scanner whose findings are read here, against the rule the module header otherwise holds, and the header now states the exception: the alternative was running it for nothing. The count is reported, never recomputed. The same report carries the could-not-look, since a dependency it could not download populates errors and drops results while still exiting 0.

The other four

scanner	what its exit code confuses	discriminator
cargo-vet	255 is an unvetted dependency or a store it could not open	a run that reported prints on stdout
cargo-deny	bitmask; 1 is a matched advisory or a database it could not fetch	same stdout test
osv-scanner	separates them itself at 127/128, both read as refusals	only 1 is a finding
zizmor	11-14 are findings by severity; exit 0 when it skips an unparseable workflow	failed to parse input: on stderr

zizmor's is the subtle one. Handed one workflow it cannot parse alongside workflows it can, it skips that one, audits the rest, reports no findings and exits 0. Its SARIF asserts executionSuccessful: true in exactly that case, so the structured output is worse than useless. This section hands zizmor a list of files, which is precisely the shape that triggers it.

tool_read() now hands each scanner its exit code, stdout and stderr and takes a reader that may name a could-not-look. The reader is consulted before the zero is believed, because two of these failures wear exit 0. A tool whose exit code already separates the two passes a reader that never fires.

Evidence

Every contract above was measured by running the installed tool, not by reading its documentation. Versions: osv-scanner 2.5.1, zizmor 1.30.0, cargo-deny 0.20.2, cargo-vet 0.10.2, guarddog 3.2.0.

Eleven new tests, each with a control. Every reader was reverted in turn to confirm the test covering it fails without it: three fail without the guarddog risks read, four without the other readers, and the controls holding that a genuine finding stays a finding pass in both directions.

A note on the seven tests this changed

Seven existing tests had guarddog stubs answering a bare exit 0. That stopped modelling the tool once the invocation gained --output-format json, because a real clean run prints a report whose risks list is empty. Their stubs were updated, not the code they test.

One of them also grepped "$3" for the requirements-file path, which the new flag shifts to "$5". It would have kept passing while asserting nothing.

ADR 0007

Records the orchestrator survey this came out of: whether supply-chain should be retired in favour of MegaLinter, trunk, SARIF aggregation, Scorecard, Trivy or pre-commit. None can express the contract, and no candidate reaches more than two of the five scanners. The four scanner defects above were the survey's actual yield.

principles/explicit-unknown.toml moves its XCCDF citation from r3 to r4 and gains the scoring asymmetry verdict() implements.

---

cargo clippy --all-targets clean under pedantic, cargo fmt --check clean, all suites pass, uphold scan passes.

https://claude.ai/code/session_01YE1DDjVegJr2MZ12TjPW6e

[coderabbitai]

Warning

Review limit reached

Next included review available in 26 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 2988fca9-7f9e-463e-8f7b-13c377b819a8

📥 Commits

Reviewing files that changed from the base of the PR and between fc78b99 and 3b3a700.

📒 Files selected for processing (5)

• docs/REFERENCE.md
• docs/adr/0007-what-the-surveyed-orchestrators-collapse.md
• principles/explicit-unknown.toml
• src/supply.rs
• tests/supply_chain_cli.rs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 88.73239% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.21%. Comparing base (8955289) to head (3b3a700).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/supply.rs	88.73%	16 Missing ⚠️

❌ Your patch status has failed because the patch coverage (88.73%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #156      +/-   ##
==========================================
- Coverage   93.26%   93.21%   -0.06%     
==========================================
  Files          38       38              
  Lines       15276    15396     +120     
==========================================
+ Hits        14247    14351     +104     
- Misses       1029     1045      +16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.