Repository navigation
Conversation
added 7 commits
October 31, 2026 18:49
…ning - Pin NVIDIA SkillSpector to the exact commit of v2.13.2 (fetched by SHA, not tag/main) and install it from that ref — a moving upstream can no longer weaken the advisory gate silently. - Add a BLOCKING gitleaks workflow (PR diff, push-to-main, weekly full-history cron), with the action pinned by SHA like SkillSpector. - Add .gitleaks.toml allowlists audited against the first scan: 60 findings, 0 real secrets (public Supabase anon JWTs shipped in client bundles, Apify actor IDs, tokenizer SHA-256 hashes, synthetic_* fixtures). Every entry documents WHY it is safe to expose.
The production ring shipped with "keys": [] — and with an empty ring both signRelease() and verifyRelease() operated anchor-less, so ANY public key passed verification. "Verified by SuperAgentSkill" was forgeable. - Register the produced Ed25519 key (id 25de98852a1efe47) via the existing manage-signing-keys.ts flow (private half stays outside the repo, 0600). - signRelease/verifyRelease and plugin signing now refuse to operate without an active key in the deployment ring; plugin downloads degrade to an explicit unsigned_reason instead of emitting an orphan signature. - Keep ring injectable for tests/BYO deployments (additive optional arg). - Add scripts/check-signing-keyrings.ts (CI-blocking hygiene: exactly one active key per ring, fingerprint match) and docs/SIGNING-KEYS.md runbook for rotation/revocation/offline verification. - Tests cover: foreign-key forgery rejected, empty ring is an error not a pass, and env keys outside the ring sign nothing.
Imports the SkillEvaluator protocol so a benchmark can no longer grade noise: - pairedComparison now emits metric_set_version + score_policy_version, a verdict (PASS >= +5pp, FAIL <= -10pp, LIMITED in between, INCONCLUSIVE when the arms are indistinguishable or the suite is under 5 cases) and a per-case basis ledger (EXCLUDED_ONLY_IN_BASELINE / ONLY_IN_CERTIFIED / JUDGE_MIXED). "Skill never activated" is a first-class INCONCLUSIVE, never a free PASS. - Judge identity is recorded per arm; mixing judge models degenerates to INCONCLUSIVE instead of silently comparing two experiments. - New src/lib/adversarial/stamping.ts: an eval run carries an evidence stamp with the required checks (full_suite, holdout_included, scan_clean, judge_source_recorded). eval-adversarial.ts writes judge_model and the stamp (scan_clean comes from the SAME blocking audit gate); a sampled run can no longer self-certify. - Trust score (both scorers) gains a versioned policy, retracts adversarial evidence when the stamp is present but not certify-worthy, and caps the score at 0.69 when the security scan did not complete — an incomplete analysis can never render green, and never verifies.
- Extract the package audit into src/lib/security/package-audit.ts so the CI
gate and runtime callers share ONE implementation (scripts/audit-skills.ts
is now only the file-walking CLI; rejectAt is an option).
- Add NVIDIA SkillSpector golden patterns to the production guard:
P2 hidden Unicode TAG instructions (U+E0000..U+E007F), MP1-MP3 memory
poisoning (credential seeding is critical), TP3 parameter-description
smuggling, AR1-AR3 unconditional anti-refusal overrides.
- Add golden pattern SC9 (.pyc/.pyo bytecode) to the malicious-function rules.
- Sanitized output now strips the tag block too; the AR rule is scoped to
unconditional refusal wipes so benign guidance ("never refuse outright")
does not false-positive.
- Golden test cases pin each pattern plus benign look-alikes; the repo's own
content still audits clean (0 packages blocked).
Port of SkillSpector's --mcp-registry-compare idea to the registry: a package that keeps its identity (slug + version + publisher) while its content or tool surface changes is the rug-pull attack, and version strings alone cannot see it. - src/lib/marketplace/rug-pull.ts: pure comparator over published vs candidate snapshots — CONTENT_CHANGED_SAME_VERSION (critical), TOOL_SET_CHANGED, PUBLISHER_CHANGED, VERSION_ROLLBACK. A new version with new content is a release, not a finding. - scripts/check-rug-pull.ts: compares a published manifest against content/skills (or --write-manifest to snapshot at release time); exit 1 on findings with --fail-on-findings. - Tests cover the attack, the legitimate release, order-insensitive tool sets and deterministic content hashing.
…rver The MCP endpoint is the product's most public contract and had zero coverage; src/lib/mcp/tools/skills.ts alone is ~2.8k lines. - tests/mcp-server.test.ts: hermetic end-to-end contract tests (no network, no Supabase) — initialize handshake, unique/complete tools/list with inputSchema, free tools anonymous, anonymous write rejected with a matched JSON-RPC id, forged bearer fail-closed with X-MCP-Auth diagnostics, GET info + CORS, and security_scan behaviour. A tiny alias loader teaches plain node the tsconfig "@/" path and extensionless relative imports. - src/lib/mcp/tools/security.ts: security_scan runs the SAME blocking gate as publish time (package-audit) and returns safe_to_install, verdict, worst severity and per-finding evidence (field/offset/excerpt), with explicit analysis-scope limits — a clean scan means 'nothing detected', not 'proven safe'. Free and anonymous: safety is not paywalled. - Register it in the server and FREE_TOOLS; add --test-force-exit to the test scripts because MCP streams remain open by design, plus npm scripts for the new suites and check:rug-pull.
…rift - check-signing-keyrings.ts now uses node:crypto/node:fs only: it runs in the gitleaks job, which has no node_modules, so importing the zod-based keyring parser failed there. Same checks (spec, 16-char id == spki fingerprint, statuses, exactly one active key, revoked keys need a reason). - Regenerate cli/super-agent.js: the CLI bundles the prompt-injection guard, so the new SkillSpector golden patterns are part of its output (--check was correctly refusing the stale bundle). - Prettier drift in four pre-existing files (payments.functions, pricing-copy, routes/pricing, its test) broke the lint gate before this branch; formatting only, no behaviour change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Por que
Auditoria de segurança do próprio repo (o pitch do produto é segurança) + estudo dos dois projetos oficiais da NVIDIA — SkillEvaluator e SkillSpector — para trazer o que eles fazem de melhor.
O que muda
1. Scanning (CI)
main). Instalação viagit fetch <sha>+checkout FETCH_HEAD.gitleaks) em PR, push namaine cron semanal de histórico completo, com a action pinada por SHA..gitleaks.tomlcom allowlists auditadas: primeiro scan → 60 achados → 0 segredos reais (JWTsrole:anondo Supabase que já vão no bundle, actor IDs da Apify, hashes SHA-256 de tokenizer, fixturessynthetic_*). Cada entrada documenta o porquê.2. Keyring de assinatura (bug crítico)
O ring de produção estava vazio (
"keys": []) e, com isso,verifyRelease()aceitava qualquer chave pública — "verified by SuperAgentSkill" era forjável.manage-signing-keys.ts rotate); privada fora do repo, 0600.unsigned_reasonexplícito; chave fora do ring é rejeitada.scripts/check-signing-keyrings.tsbloqueante no CI (exatamente 1 chave ativa por ring, fingerprint confere) + runbookdocs/SIGNING-KEYS.md.3. Protocolo de eval (SkillEvaluator)
metric_set_version+score_policy_versionem todo benchmark pareado.PASS≥ +5pp,FAIL≤ −10pp,LIMITEDno meio,INCONCLUSIVEquando os braços são indistinguíveis (a skill nunca ativou) ou a suíte tem < 5 casos — nunca PASS de graça.basis: ledger por case comEXCLUDED_ONLY_IN_BASELINE,EXCLUDED_ONLY_IN_CERTIFIED,EXCLUDED_JUDGE_MIXED.INCONCLUSIVE.src/lib/adversarial/stamping.ts):full_suite,holdout_included,scan_clean(o MESMO gate bloqueante),judge_source_recorded. Run amostrado não auto-certifica.verified— análise incompleta nunca fica verde (nos dois scorers,score.tsescoring.ts).4. Golden patterns do SkillSpector no gate
.pyc/.pyo.src/lib/security/package-audit.ts(uma implementação para o gate CI e runtime);scripts/audit-skills.tsvira só o CLI.5. MCP
tests/mcp-server.test.ts): handshake, catálogo único cominputSchema, tool grátis anônima, write anônima → 401 com id ecoado, bearer forjado fail-closed comX-MCP-Auth, GET info + CORS. Loader de alias@/em node puro.security_scan: scanner como guardrail de runtime, grátis/anônimo, roda o MESMO gate do publish e devolvesafe_to_install/verdict + evidência por finding (campo/offset/excerpt), com limites de análise explícitos.6. Rug-pull detection
src/lib/marketplace/rug-pull.ts+scripts/check-rug-pull.ts: mesmo slug+versão com conteúdo ou tool surface diferente = rug pull (crítico); publisher trocado e rollback de versão também. Release legítima (versão nova) não é finding.Verificação
npm run test:all→ 410/410 pass, 0 failnpx tsc --noEmit→ limpogitleaks detect→ no leaks foundaudit:skills→ 0 pacotes bloqueadoscheck-signing-keyrings→ ambos os rings okOs secrets
SIGNING_PRIVATE_KEY/SIGNING_PUBLIC_KEY(GitHub +.env) precisam ser o par registrado no ring (id25de98852a1efe47), senão as assinaturas de plugin/trust degradam paraunsigned(fail-closed) até a rotação. Os PEMs estão em~/.super-agent-skill/keys/.