From 13e5442a16e37d65cf24ff13582f5adea1da958f Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Thu, 30 Jul 2026 11:59:21 -0400 Subject: [PATCH 1/4] test(hosted): production e2e for `scan --mode hosted` across every ecosystem and package manager MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds `crates/socket-patch-cli/tests/e2e_hosted_production.rs`: the first suite that exercises hosted mode against the REAL production service with no mocking anywhere. Every other hosted-mode capstone (`e2e_redirect_*_build.rs`) serves the patched artifact from a local wiremock, which proves the rewrite grammar but cannot notice production drifting away from the CLI. Each leg installs a pinned vulnerable dependency from its real upstream registry, asserts the bytes are pristine, runs `scan --mode hosted`, asserts the lockfile now pins patch.socket.dev, then WIPES the install tree and reinstalls from the rewritten lock alone — so the package manager itself fetches from Socket and verifies the integrity pin it was handed. That last step is the point: it is the only place in the repo where a third-party package manager, not socket-patch, downloads a Socket-hosted artifact and independently checks its checksum. Coverage — 14 legs, all green: npm package-lock, npm-shrinkwrap, pnpm, yarn classic, yarn berry, bun PyPI requirements.txt, uv.lock Cargo per-patch sparse registry gem redirect asserted (install blocked by a server defect, below) golang documented refusal (negative assertion) deno unsupported (negative assertion) preflight canary that the four required patches are still published canary maven/nuget/composer, which have zero free published patches today Runs against the unauthenticated public proxy on purpose — no API token is used or needed, and SOCKET_API_TOKEN is scrubbed from the child environment. CI: new `hosted-e2e` job intended as a required status check. It has no job-level `if:`, no `needs:`, no matrix and no continue-on-error, because a SKIPPED required check can wedge a PR at "Expected — waiting for status"; the kill switch gates the steps, never the job. Set the repo variable HOSTED_E2E_DISABLED=true and re-run to bypass without a commit, or use the new workflow_dispatch `hosted_e2e` input for a one-off force/skip. The suite stays `#[ignore]`-gated, so the existing `test` and `e2e` jobs are unaffected. Three real issues surfaced by running against production, documented in docs/testing/hosted-production-e2e.md: * gem hosted mode is unusable for gems with dependencies — the Socket gem patch-registry compact index omits runtime deps, so bundler fails closed with APIResponseMismatchError (SERVER fix) * pnpm 11 rejects hosted lockfiles by default with ERR_PNPM_TARBALL_URL_MISMATCH; `--trust-lockfile` is the opt-out and the CLI should warn about it (CLI UX fix) * the uv.lock rewriter points the `sdist` entry at the patched wheel while keeping the sdist's original `size` (CLI, minor) Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 173 ++ .../tests/e2e_hosted_production.rs | 1510 +++++++++++++++++ docs/testing/hosted-production-e2e.md | 260 +++ 3 files changed, 1943 insertions(+) create mode 100644 crates/socket-patch-cli/tests/e2e_hosted_production.rs create mode 100644 docs/testing/hosted-production-e2e.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 39c56ba..95cad51 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -4,6 +4,15 @@ on: push: branches: [main] pull_request: + workflow_dispatch: + inputs: + hosted_e2e: + # Underscored on purpose: `inputs.hosted-e2e` is not valid expression + # syntax (a hyphenated name needs `inputs['hosted-e2e']`). + description: 'hosted-e2e: auto (obey vars.HOSTED_E2E_DISABLED) | force | skip' + type: choice + default: auto + options: [auto, force, skip] permissions: contents: read @@ -755,4 +764,168 @@ jobs: with: name: setup-matrix-${{ matrix.ecosystem }} path: report-${{ matrix.ecosystem }}.json + + # ---------------------------------------------------------------------- + # Hosted-mode production e2e — REQUIRED status check, with a kill switch. + # + # Drives `scan --mode hosted` against the REAL production endpoints + # (patches-api.socket.dev + patch.socket.dev) and the REAL upstream + # registries, using patches that are actually published on production. + # Nothing is mocked. Every other hosted-mode capstone in this repo + # (e2e_redirect_*) points at a wiremock stand-in, so this job is the only + # thing that would notice production drifting away from the CLI. + # + # The suite itself is `#[ignore]`-gated, so it stays OUT of the `test` and + # `e2e` jobs and only runs where it is explicitly asked for — here. + # + # INVARIANTS (this job is registered in branch protection as a required + # check named exactly `hosted-e2e`): + # * NO job-level `if:` — a *skipped* required check is ambiguous to branch + # protection and can wedge a PR at "Expected — waiting for status". + # The kill switch gates the STEPS, never the job. + # * NO `needs:` — an upstream failure would skip this job, same wedge. + # * NO matrix and NO rename — the check name must stay `hosted-e2e`. + # * NO `continue-on-error` — a bypass must be visible, not invisible. + # The job ALWAYS runs and ALWAYS reaches success or failure. + # + # ESCAPE HATCH — when production is down and this is blocking merges: + # Settings -> Secrets and variables -> Actions -> Variables -> + # HOSTED_E2E_DISABLED = true + # then "Re-run failed jobs" on any blocked PR. `vars` is read at job-run + # time, so no commit and no push is needed; the job goes green with a loud + # ::warning:: and a BYPASSED banner in the job summary. DELETE the variable + # to re-arm. For a one-off: Actions -> CI -> Run workflow -> + # hosted_e2e = force (ignore the variable) | skip (bypass this run). + # ---------------------------------------------------------------------- + hosted-e2e: + name: hosted-e2e # registered in branch protection; do not rename + runs-on: ubuntu-latest + permissions: + contents: read + timeout-minutes: 30 + concurrency: + # These are real requests against a real production service — keep it to + # one run per ref rather than one per push. + group: hosted-e2e-${{ github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + env: + HOSTED_E2E_DISABLED: ${{ vars.HOSTED_E2E_DISABLED }} + # The `inputs` context is empty on push/pull_request, so default to auto. + HOSTED_E2E_MODE: ${{ (github.event_name == 'workflow_dispatch' && inputs.hosted_e2e) || 'auto' }} + steps: + - name: Resolve the kill switch + id: gate + run: | + set -eu + run=true; reason='' + case "$HOSTED_E2E_MODE" in + force) reason='workflow_dispatch hosted_e2e=force (kill switch ignored)' ;; + skip) run=false; reason='workflow_dispatch hosted_e2e=skip' ;; + *) if [ "${HOSTED_E2E_DISABLED:-}" = 'true' ]; then + run=false + reason='repository variable HOSTED_E2E_DISABLED=true' + fi ;; + esac + echo "run=$run" >> "$GITHUB_OUTPUT" + if [ "$run" = 'false' ]; then + echo "::warning title=hosted-e2e BYPASSED::$reason" + { + echo '## :warning: hosted-e2e BYPASSED — no production coverage in this run' + echo + echo "Reason: $reason" + echo + echo 'Re-arm by deleting the HOSTED_E2E_DISABLED repository variable' + echo '(Settings -> Secrets and variables -> Actions -> Variables).' + } >> "$GITHUB_STEP_SUMMARY" + fi + + - name: Checkout + if: steps.gate.outputs.run == 'true' + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + persist-credentials: false + + - name: Install Rust + if: steps.gate.outputs.run == 'true' + run: rustup show + + - name: Cache cargo + if: steps.gate.outputs.run == 'true' + uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2.9.1 + with: + key: hosted-e2e + save-if: ${{ github.ref == 'refs/heads/main' }} + + - name: Setup Node.js + if: steps.gate.outputs.run == 'true' + uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 + with: + node-version: '20.20.2' + + - name: Setup npm-family package managers + if: steps.gate.outputs.run == 'true' + # corepack resolves the pinned pnpm / yarn-classic / yarn-berry the + # suite's fixtures request via their `packageManager` field. bun comes + # from npm for the same reason pnpm does in the `e2e` job: it is the + # one install path that works uniformly across runners. + run: | + set -eu + corepack enable + npm install -g bun@1 + node --version && npm --version && bun --version + + - name: Setup Python + uv + if: steps.gate.outputs.run == 'true' + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 + with: + python-version: '3.12.x' + + - name: Install uv + if: steps.gate.outputs.run == 'true' + run: python -m pip install --disable-pip-version-check uv && uv --version + + - name: Setup Ruby + if: steps.gate.outputs.run == 'true' + uses: ruby/setup-ruby@319994f95fa847cf3fb3cd3dbe89f6dcde9f178f # v1.295.0 + with: + ruby-version: '3.2.10' + # The gem hosted rewrite pins into the Gemfile.lock CHECKSUMS + # section, which `bundle lock --add-checksums` only emits on >= 2.6. + bundler: '2.6' + bundler-cache: false + + - name: Setup Go + if: steps.gate.outputs.run == 'true' + uses: actions/setup-go@d35c59abb061a4a6fb18e82ac0862c26744d6ab5 # v5 + with: + go-version: '1.23' + + - name: Run hosted-mode production e2e + if: steps.gate.outputs.run == 'true' + env: + # A required check must never report green on an unexercised leg: + # STRICT turns the suite's local "toolchain missing" soft-skips into + # hard failures. Every toolchain it needs is installed above. + SOCKET_PATCH_HOSTED_E2E_STRICT: '1' + run: | + set -u + # The public proxy intermittently returns 503 "Service temporarily + # over capacity" — that is the documented reason the older live-API + # suites were pulled from the PR matrix (see the `e2e` job). Retry the + # whole suite a couple of times before calling it a real failure, so a + # transient 503 does not block merges through a required check. + for attempt in 1 2 3; do + echo "::group::hosted-e2e attempt $attempt" + cargo test -p socket-patch-cli --test e2e_hosted_production -- \ + --ignored --nocapture --test-threads=4 + status=$? + echo "::endgroup::" + if [ "$status" -eq 0 ]; then + exit 0 + fi + echo "::warning title=hosted-e2e attempt $attempt failed::retrying" + sleep $((attempt * 20)) + done + echo "::error title=hosted-e2e::suite failed on all 3 attempts" + exit 1 if-no-files-found: warn diff --git a/crates/socket-patch-cli/tests/e2e_hosted_production.rs b/crates/socket-patch-cli/tests/e2e_hosted_production.rs new file mode 100644 index 0000000..bda0ca9 --- /dev/null +++ b/crates/socket-patch-cli/tests/e2e_hosted_production.rs @@ -0,0 +1,1510 @@ +//! Hosted-mode (`scan --mode hosted`) end-to-end tests against **production**. +//! +//! Every other hosted-mode capstone in this repo (`e2e_redirect_*_build.rs`) +//! points the CLI at a wiremock stand-in for `patch.socket.dev`. This suite is +//! the opposite: it contacts the **real** Socket production endpoints and the +//! **real** upstream registries, with **no mocking anywhere**, and proves the +//! full hosted loop for each ecosystem and package manager: +//! +//! 1. install a pinned, known-vulnerable dependency with a real package +//! manager, from its real upstream registry; +//! 2. assert the installed bytes are **pristine** (anti-vacuity — without +//! this every "patched" assertion below could pass on a no-op); +//! 3. run `socket-patch scan --mode hosted --json --yes`, which resolves a +//! hosted patch reference from `patches-api.socket.dev` and rewrites the +//! lockfile / registry config to point at `patch.socket.dev`; +//! 4. assert the rewrite landed (host + patch UUID present in the lock, and +//! the integrity pin was replaced); +//! 5. **wipe the install tree and reinstall from the rewritten lock alone**, +//! letting the package manager itself fetch from `patch.socket.dev` and +//! verify the integrity pin it was given; +//! 6. assert the reinstalled bytes now carry the patch. +//! +//! Step 5 is the point of the suite. It is the only test in the repo where a +//! third-party package manager — not socket-patch — downloads a Socket-hosted +//! artifact and independently verifies its checksum. +//! +//! # Required production patches +//! +//! These tests are pinned to specific patches that must stay published and +//! **free-tier** on `patches-api.socket.dev`. If Socket unpublishes one, the +//! `preflight_required_patches_are_published` test fails first and names it, +//! rather than letting a downstream leg fail with a confusing symptom. +//! +//! | Ecosystem | PURL | Patch UUID | Advisory | +//! |-----------|------|------------|----------| +//! | npm | `pkg:npm/minimist@1.2.2` | `80630680-4da6-45f9-bba8-b888e0ffd58c` | GHSA-xvch-5gv4-984h (CVE-2021-44906) | +//! | PyPI | `pkg:pypi/urllib3@1.26.18` | *any of three* (see [`PYPI_UUIDS`]) | GHSA-gm62-xv2j-4w53 &co | +//! | Cargo | `pkg:cargo/traitobject@0.1.1` | `cf2e6f58-d9fa-4096-9151-c34afa717f89` | GHSA-pp8r-vv2j-9j5v | +//! | gem | `pkg:gem/activestorage@7.0.2.2` | `2535d43d-67ce-4944-be27-c19e113997fb` | GHSA-w749-p3v6-hccq | +//! +//! `docs/testing/hosted-production-e2e.md` explains how these were chosen and +//! how to re-pick one if it is ever withdrawn. +//! +//! # Ecosystems with no coverage, and why +//! +//! * **maven / nuget / composer** — hosted mode is implemented and documented +//! for all three, but production currently publishes **zero** free-tier +//! patches for them, so there is nothing real to redirect to. Rather than +//! silently skipping, [`canary_unpublished_ecosystems`] probes production +//! every run and tells us the moment that changes. +//! * **golang** — hosted mode is refused **by design** +//! (`docs/design/golang-hosted-no-go.md`). Covered as a negative assertion. +//! * **deno** — hosted mode is not supported. Covered as a negative assertion. +//! +//! # Prerequisites +//! +//! Toolchains (each leg soft-skips if its own toolchain is absent, unless +//! `SOCKET_PATCH_HOSTED_E2E_STRICT=1`): `npm`, `pnpm`, `yarn` (classic), +//! `corepack` (berry), `bun`, `uv`, `cargo`, `ruby` + `bundle`, `go`. +//! +//! Network egress to: `patches-api.socket.dev`, `patch.socket.dev`, +//! `registry.npmjs.org`, `pypi.org`, `files.pythonhosted.org`, +//! `static.crates.io`, `index.crates.io`, `rubygems.org`. +//! +//! No API token is used or needed — the suite deliberately runs against the +//! **free public proxy**, which is the surface every unauthenticated user +//! gets. `SOCKET_API_TOKEN` is scrubbed from the child environment. +//! +//! # Running +//! +//! ```sh +//! cargo test -p socket-patch-cli --test e2e_hosted_production -- --ignored +//! +//! # CI (required job): turn every soft-skip into a hard failure, so a missing +//! # toolchain can never report green on a required check. +//! SOCKET_PATCH_HOSTED_E2E_STRICT=1 \ +//! cargo test -p socket-patch-cli --test e2e_hosted_production -- --ignored +//! ``` + +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +use socket_patch_cli::args::{GLOBAL_ARG_ENV_VARS, LOCAL_ARG_ENV_VARS}; + +// --------------------------------------------------------------------------- +// Production endpoints + required-patch catalog +// --------------------------------------------------------------------------- + +/// The free public patch proxy. Deliberately hard-coded rather than read from +/// the environment: this suite's entire purpose is to exercise *production*, +/// and an ambient `SOCKET_PROXY_URL` pointing at staging would let it pass +/// while proving nothing. +const PROXY: &str = "https://patches-api.socket.dev"; + +/// The host every hosted-mode rewrite must point the package manager at. +const PATCH_HOST: &str = "patch.socket.dev"; + +const NPM_PURL: &str = "pkg:npm/minimist@1.2.2"; +const NPM_NAME: &str = "minimist"; +const NPM_VERSION: &str = "1.2.2"; +const NPM_UUID: &str = "80630680-4da6-45f9-bba8-b888e0ffd58c"; + +const PYPI_PURL: &str = "pkg:pypi/urllib3@1.26.18"; +const PYPI_NAME: &str = "urllib3"; +const PYPI_VERSION: &str = "1.26.18"; +/// urllib3 1.26.18 carries **three** distinct free patches (one per advisory). +/// Which one the resolver selects is a server-side ordering detail, so the +/// tests assert "one of these" rather than pinning a single UUID — pinning one +/// would make the suite red on an unrelated server-side reorder. +const PYPI_UUIDS: &[&str] = &[ + "de58c8b8-796c-4b6d-8a48-539b5563db76", + "26242e35-f867-4da8-8789-f0d2ea49e0f1", + "e828efa5-5c6d-43f3-9909-03f5ac232b98", +]; + +const CARGO_PURL: &str = "pkg:cargo/traitobject@0.1.1"; +const CARGO_NAME: &str = "traitobject"; +const CARGO_VERSION: &str = "0.1.1"; +const CARGO_UUID: &str = "cf2e6f58-d9fa-4096-9151-c34afa717f89"; +/// The traitobject patch annotates `src/lib.rs` with its advisory ID (the +/// crate is unmaintained; the patch documents that and fixes deprecations). +/// Cargo crates are not rewritten with the `// Socket Community Patch` header +/// that npm/PyPI artifacts carry, so this is the marker to look for. +const CARGO_MARKER: &str = "GHSA-pp8r-vv2j-9j5v"; + +const GEM_PURL: &str = "pkg:gem/activestorage@7.0.2.2"; +const GEM_NAME: &str = "activestorage"; +const GEM_VERSION: &str = "7.0.2.2"; +const GEM_UUID: &str = "2535d43d-67ce-4944-be27-c19e113997fb"; + +/// Header the patch service injects into patched npm / PyPI source files. +const PATCH_MARKER: &str = "Socket Community Patch"; + +/// Ecosystems where hosted mode is implemented but production has no free +/// patches to exercise it with. [`canary_unpublished_ecosystems`] watches +/// these so coverage can be extended the moment one lights up. +const UNPUBLISHED_ECOSYSTEMS: &[(&str, &[&str])] = &[ + ( + "maven", + &[ + "pkg:maven/org.apache.logging.log4j/log4j-core", + "pkg:maven/com.fasterxml.jackson.core/jackson-databind", + "pkg:maven/org.yaml/snakeyaml", + "pkg:maven/commons-io/commons-io", + ], + ), + ( + "nuget", + &[ + "pkg:nuget/Newtonsoft.Json", + "pkg:nuget/System.Text.Json", + "pkg:nuget/SharpZipLib", + "pkg:nuget/RestSharp", + ], + ), + ( + "composer", + &[ + "pkg:composer/guzzlehttp/guzzle", + "pkg:composer/symfony/http-kernel", + "pkg:composer/laravel/framework", + "pkg:composer/monolog/monolog", + ], + ), +]; + +// --------------------------------------------------------------------------- +// Strictness + skip policy +// --------------------------------------------------------------------------- + +/// In CI this suite backs a **required** status check, so a leg that quietly +/// returns early because a toolchain is missing would report green while +/// proving nothing. `SOCKET_PATCH_HOSTED_E2E_STRICT=1` converts every soft +/// skip into a hard failure. Locally it stays off so a developer without, +/// say, `bun` can still run the rest. +fn strict() -> bool { + std::env::var("SOCKET_PATCH_HOSTED_E2E_STRICT") + .map(|v| v == "1" || v.eq_ignore_ascii_case("true")) + .unwrap_or(false) +} + +/// Soft-skip a leg: panics under [`strict`], otherwise prints a tagged notice +/// and returns from the calling test. +macro_rules! soft_skip { + ($leg:expr, $($arg:tt)*) => {{ + let why = format!($($arg)*); + if strict() { + panic!( + "STRICT: {} cannot run: {why}\n\ + SOCKET_PATCH_HOSTED_E2E_STRICT=1 forbids skipping — a required \ + CI check must never report green on an unexercised leg. Install \ + the missing toolchain, or unset the strict flag for local runs.", + $leg + ); + } + println!("SKIP {}: {why}", $leg); + return; + }}; +} + +// --------------------------------------------------------------------------- +// CLI invocation +// --------------------------------------------------------------------------- + +fn binary() -> PathBuf { + env!("CARGO_BIN_EXE_socket-patch").into() +} + +fn has_command(cmd: &str) -> bool { + // `go` has no `--version` — it takes `go version` as a subcommand and + // errors with "flag provided but not defined: -version" otherwise. Probing + // it the usual way silently skips the golang leg on a machine that has Go. + let probe: &[&str] = if cmd == "go" { + &["version"] + } else { + &["--version"] + }; + Command::new(cmd) + .args(probe) + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .status() + .map(|s| s.success()) + .unwrap_or(false) +} + +/// The three legacy `SOCKET_PATCH_*` names still honored at runtime via +/// `socket_patch_core::env_compat` — not in the clap-bound lists, so they need +/// scrubbing separately. +const LEGACY_ENV_VARS: &[&str] = &[ + "SOCKET_PATCH_PROXY_URL", + "SOCKET_PATCH_DEBUG", + "SOCKET_PATCH_TELEMETRY_DISABLED", +]; + +/// Run the CLI with a hermetically pinned environment. +/// +/// The scrub matters more here than in any offline suite. An ambient +/// `SOCKET_PROXY_URL` or `SOCKET_API_URL` would silently point the run at +/// staging — and every assertion below would still pass, proving nothing about +/// production. An ambient `SOCKET_API_TOKEN` would move the run off the free +/// public proxy that this suite exists to cover. The hostile seeds below are +/// removed by the same loop that removes the real ones, so if the scrub is +/// ever dropped the seeds turn the suite red immediately instead of letting a +/// developer's ambient shell decide what got tested. +fn run(cwd: &Path, args: &[&str]) -> (i32, String, String) { + let mut cmd = Command::new(binary()); + cmd.args(args) + .current_dir(cwd) + .env("SOCKET_GLOBAL", "true") + .env("SOCKET_GLOBAL_PREFIX", "/nonexistent") + .env("SOCKET_DRY_RUN", "true") + .env("SOCKET_SAVE_ONLY", "true") + .env("SOCKET_OFFLINE", "true") + .env("SOCKET_API_TOKEN", "hostile-seed-must-be-scrubbed") + .env("SOCKET_PROXY_URL", "http://127.0.0.1:1/hostile") + .env("SOCKET_MANIFEST_PATH", "/nonexistent/manifest.json"); + for var in GLOBAL_ARG_ENV_VARS + .iter() + .chain(LOCAL_ARG_ENV_VARS) + .chain(LEGACY_ENV_VARS) + { + cmd.env_remove(var); + } + let out: Output = cmd.output().expect("failed to execute socket-patch binary"); + ( + out.status.code().unwrap_or(-1), + String::from_utf8_lossy(&out.stdout).to_string(), + String::from_utf8_lossy(&out.stderr).to_string(), + ) +} + +/// `scan --mode hosted --json --yes` in `cwd`, asserting a clean exit and a +/// `"status": "success"` envelope. Returns the parsed envelope. +fn scan_hosted(cwd: &Path, extra: &[&str]) -> serde_json::Value { + let mut args: Vec<&str> = vec!["scan", "--mode", "hosted", "--json", "--yes"]; + args.extend_from_slice(extra); + let (code, stdout, stderr) = run(cwd, &args); + assert_eq!( + code, 0, + "scan --mode hosted failed (exit {code}).\nstdout:\n{stdout}\nstderr:\n{stderr}" + ); + let env: serde_json::Value = serde_json::from_str(&stdout).unwrap_or_else(|e| { + panic!("scan --mode hosted did not emit JSON ({e}).\nstdout:\n{stdout}\nstderr:\n{stderr}") + }); + assert_eq!( + env["status"].as_str(), + Some("success"), + // Exit 0 alone is not enough: the envelope carries the real verdict. + "scan --mode hosted did not report success.\nenvelope:\n{env:#}\nstderr:\n{stderr}" + ); + env +} + +/// Assert the hosted redirect actually rewrote something, and return the list +/// of rewritten files. +/// +/// `redirected >= 1` is the anti-vacuity guard: a run that discovered nothing +/// also exits 0 with `"status": "success"`, so without this a broken crawler +/// would look identical to a working redirect. +fn assert_redirected(env: &serde_json::Value, expect_file: &str) -> Vec { + let redirect = &env["redirect"]; + assert!( + !redirect.is_null(), + "scan --mode hosted emitted no `redirect` sub-object at all. The CLI \ + omits it entirely when discovery found nothing, so this means the \ + crawler did not see the installed dependency.\nenvelope:\n{env:#}" + ); + assert_eq!( + redirect["mode"].as_str(), + Some("hosted"), + "redirect sub-object missing or not hosted mode:\n{env:#}" + ); + let n = redirect["redirected"].as_u64().unwrap_or(0); + assert!( + n >= 1, + "hosted redirect rewrote nothing — the patch is published and the \ + package is installed, so 0 means discovery or reference resolution \ + broke.\nenvelope:\n{env:#}" + ); + let files: Vec = redirect["rewrittenFiles"] + .as_array() + .map(|a| { + a.iter() + .filter_map(|v| v.as_str().map(str::to_string)) + .collect() + }) + .unwrap_or_default(); + assert!( + files.iter().any(|f| f == expect_file), + "expected `{expect_file}` among rewrittenFiles, got {files:?}\nenvelope:\n{env:#}" + ); + files +} + +/// How many dependencies a hosted run redirected. +/// +/// `scan --mode hosted` omits the whole `redirect` sub-object when discovery +/// turned up nothing — a plain scan envelope comes back instead. For the +/// documented-unsupported ecosystems (golang, deno) "no redirect object" and +/// `"redirected": 0` are the same verdict, so normalize them. +fn redirected_count(env: &serde_json::Value) -> u64 { + let redirect = &env["redirect"]; + if redirect.is_null() { + return 0; + } + redirect["redirected"].as_u64().unwrap_or(0) +} + +// --------------------------------------------------------------------------- +// Toolchain invocation +// --------------------------------------------------------------------------- + +/// Run an external package manager. Returns the `Output` without asserting, so +/// callers can distinguish "the registry was unreachable" (soft-skip material +/// during fixture setup) from "the install of the redirected lock failed" +/// (always a hard failure — that is the thing under test). +fn tool(cwd: &Path, program: &str, args: &[&str], env: &[(&str, &str)]) -> Output { + let mut cmd = Command::new(program); + cmd.args(args).current_dir(cwd); + // Keep every toolchain's cache inside the fixture so the reinstall leg + // starts genuinely cold and cannot be satisfied from a warm host cache + // holding the *pristine* artifact. + for (k, v) in env { + cmd.env(k, v); + } + // A `VIRTUAL_ENV` inherited from the developer's shell makes uv install + // into the wrong interpreter. + cmd.env_remove("VIRTUAL_ENV"); + cmd.output() + .unwrap_or_else(|e| panic!("failed to spawn `{program}`: {e}")) +} + +fn ok(out: &Output) -> bool { + out.status.success() +} + +fn dump(out: &Output) -> String { + format!( + "exit={:?}\nstdout:\n{}\nstderr:\n{}", + out.status.code(), + String::from_utf8_lossy(&out.stdout), + String::from_utf8_lossy(&out.stderr) + ) +} + +fn read(p: &Path) -> String { + std::fs::read_to_string(p).unwrap_or_else(|e| panic!("read {}: {e}", p.display())) +} + +/// Assert `path` exists and does NOT yet carry a patch marker. +/// +/// Every "the reinstall delivered a patched artifact" assertion downstream is +/// vacuous without this: if the upstream registry ever started shipping the +/// patched bytes, or a warm cache leaked them in, the test would pass while +/// proving nothing about hosted mode. +fn assert_pristine(path: &Path, marker: &str, what: &str) { + assert!( + path.exists(), + "{what}: expected the pristine install at {} — fixture setup did not \ + produce the file under test", + path.display() + ); + let body = read(path); + assert!( + !body.contains(marker), + "{what}: the freshly-installed upstream artifact at {} ALREADY contains \ + `{marker}` before any redirect ran. Every downstream assertion would be \ + vacuous. Check for a warm package-manager cache leaking patched bytes.", + path.display() + ); +} + +fn assert_patched(path: &Path, marker: &str, what: &str) { + assert!( + path.exists(), + "{what}: reinstall from the redirected lock did not produce {}", + path.display() + ); + let body = read(path); + assert!( + body.contains(marker), + "{what}: reinstalled from the redirected lock, but {} does not contain \ + `{marker}` — the package manager fetched something, and it was not the \ + patched artifact.", + path.display() + ); +} + +/// Assert a rewritten lockfile points at the hosted patch server for the +/// expected patch. +/// +/// Deliberately does NOT assert the grant token embedded in the URL: the +/// service mints a fresh one per reference request, so pinning it would make +/// the suite red on the second run. +fn assert_hosted_pin(lock_body: &str, uuids: &[&str], what: &str) { + assert!( + lock_body.contains(PATCH_HOST), + "{what}: rewritten lock does not reference {PATCH_HOST}:\n{lock_body}" + ); + assert!( + uuids.iter().any(|u| lock_body.contains(u)), + "{what}: rewritten lock references {PATCH_HOST} but carries none of the \ + expected patch UUIDs {uuids:?} — the redirect resolved a different \ + patch than the catalog pins.\n{lock_body}" + ); +} + +// --------------------------------------------------------------------------- +// Production reachability probes (used by the preflight + canary tests) +// --------------------------------------------------------------------------- + +/// `GET /patch/by-package/` against the real proxy. Returns the patch +/// UUIDs published for `purl`, or an `Err` describing a transport failure. +async fn published_uuids(purl: &str) -> Result, String> { + let url = format!("{PROXY}/patch/by-package/{}", urlencode(purl)); + let resp = reqwest::Client::new() + .get(&url) + .header("Accept", "application/json") + .send() + .await + .map_err(|e| format!("GET {url}: {e}"))?; + let status = resp.status(); + let body = resp + .text() + .await + .map_err(|e| format!("GET {url}: reading body: {e}"))?; + if !status.is_success() { + return Err(format!("GET {url}: HTTP {status}\n{body}")); + } + let v: serde_json::Value = + serde_json::from_str(&body).map_err(|e| format!("GET {url}: bad JSON ({e}):\n{body}"))?; + Ok(v["patches"] + .as_array() + .map(|a| { + a.iter() + .filter_map(|p| p["uuid"].as_str().map(str::to_string)) + .collect() + }) + .unwrap_or_default()) +} + +/// Percent-encode a PURL for use as a single path segment. `reqwest` will not +/// do this for us — a raw `pkg:npm/...` would be split into path segments and +/// 404. +fn urlencode(s: &str) -> String { + let mut out = String::with_capacity(s.len() * 3); + for b in s.as_bytes() { + match b { + b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'.' | b'_' | b'~' => { + out.push(*b as char) + } + _ => out.push_str(&format!("%{b:02X}")), + } + } + out +} + +// =========================================================================== +// Preflight — the catalog canary +// =========================================================================== + +/// Verify every patch this suite depends on is still published and free-tier. +/// +/// This runs first (alphabetically it sorts under `preflight_`) so that a +/// withdrawn patch produces one clear failure naming the PURL, instead of N +/// confusing downstream failures that look like CLI regressions. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "live production API: contacts patches-api.socket.dev. Run with --ignored."] +async fn preflight_required_patches_are_published() { + // (purl, acceptable uuids) + let required: Vec<(&str, Vec<&str>)> = vec![ + (NPM_PURL, vec![NPM_UUID]), + (PYPI_PURL, PYPI_UUIDS.to_vec()), + (CARGO_PURL, vec![CARGO_UUID]), + (GEM_PURL, vec![GEM_UUID]), + ]; + + let mut failures: Vec = Vec::new(); + for (purl, expected) in &required { + match published_uuids(purl).await { + Err(e) => failures.push(format!("{purl}: production probe failed: {e}")), + Ok(found) if found.is_empty() => failures.push(format!( + "{purl}: production publishes NO free patches for this package \ + anymore. This suite is pinned to it — pick a replacement and \ + update both the catalog constants in this file and \ + docs/testing/hosted-production-e2e.md." + )), + Ok(found) => { + if !expected.iter().any(|u| found.iter().any(|f| f == u)) { + failures.push(format!( + "{purl}: expected one of {expected:?} but production now \ + publishes {found:?}. The patch was replaced — update the \ + catalog constants in this file." + )); + } + } + } + } + + assert!( + failures.is_empty(), + "required production patches are no longer available:\n - {}", + failures.join("\n - ") + ); +} + +// =========================================================================== +// npm ecosystem — five package managers, five lockfile flavors +// =========================================================================== + +/// Shared npm-family fixture: a temp project with `minimist@1.2.2` pinned. +struct NpmFixture { + _tmp: tempfile::TempDir, + proj: PathBuf, + cache: PathBuf, +} + +fn npm_fixture(name: &str) -> NpmFixture { + let tmp = tempfile::tempdir().expect("tempdir"); + let proj = tmp.path().join("proj"); + let cache = tmp.path().join(format!("{name}-cache")); + std::fs::create_dir_all(&proj).expect("mkdir proj"); + std::fs::create_dir_all(&cache).expect("mkdir cache"); + // Hand-written rather than `npm init -y`, which rejects tempdir names. + std::fs::write( + proj.join("package.json"), + format!( + r#"{{"name":"hosted-e2e","version":"0.0.0","private":true,"dependencies":{{"{NPM_NAME}":"{NPM_VERSION}"}}}}"# + ), + ) + .expect("write package.json"); + NpmFixture { + _tmp: tmp, + proj, + cache, + } +} + +fn minimist_entry(proj: &Path) -> PathBuf { + proj.join("node_modules").join(NPM_NAME).join("index.js") +} + +#[test] +#[ignore = "live production API + real npm registry. Run with --ignored."] +fn npm_package_lock_hosted_install_proof() { + const LEG: &str = "npm_package_lock_hosted_install_proof"; + if !has_command("npm") { + soft_skip!(LEG, "`npm` not on PATH"); + } + let fx = npm_fixture("npm"); + let cache = fx.cache.display().to_string(); + let env = [("npm_config_cache", cache.as_str())]; + + let install = tool( + &fx.proj, + "npm", + &["install", "--no-audit", "--no-fund", "--ignore-scripts"], + &env, + ); + if !ok(&install) { + soft_skip!(LEG, "upstream `npm install` failed:\n{}", dump(&install)); + } + + assert_pristine(&minimist_entry(&fx.proj), PATCH_MARKER, LEG); + + let env_json = scan_hosted(&fx.proj, &[]); + assert_redirected(&env_json, "package-lock.json"); + + let lock = read(&fx.proj.join("package-lock.json")); + assert_hosted_pin(&lock, &[NPM_UUID], LEG); + assert!( + !lock.contains("registry.npmjs.org/minimist/-/minimist-1.2.2.tgz"), + "{LEG}: the upstream minimist tarball URL survived the rewrite — npm \ + would still install the unpatched artifact:\n{lock}" + ); + + // The proof: wipe node_modules and let npm install from the rewritten lock + // alone. npm verifies the `integrity` pin it was handed, so a success here + // means patch.socket.dev served bytes matching the hash the API published. + std::fs::remove_dir_all(fx.proj.join("node_modules")).expect("rm node_modules"); + let ci = tool( + &fx.proj, + "npm", + &["ci", "--no-audit", "--no-fund", "--ignore-scripts"], + &env, + ); + assert!( + ok(&ci), + "{LEG}: `npm ci` from the redirected lock failed — npm could not fetch \ + or could not verify the hosted artifact:\n{}", + dump(&ci) + ); + assert_patched(&minimist_entry(&fx.proj), PATCH_MARKER, LEG); +} + +#[test] +#[ignore = "live production API + real npm registry. Run with --ignored."] +fn npm_shrinkwrap_hosted_redirect() { + const LEG: &str = "npm_shrinkwrap_hosted_redirect"; + if !has_command("npm") { + soft_skip!(LEG, "`npm` not on PATH"); + } + let fx = npm_fixture("shrinkwrap"); + let cache = fx.cache.display().to_string(); + let env = [("npm_config_cache", cache.as_str())]; + + let install = tool( + &fx.proj, + "npm", + &["install", "--no-audit", "--no-fund", "--ignore-scripts"], + &env, + ); + if !ok(&install) { + soft_skip!(LEG, "upstream `npm install` failed:\n{}", dump(&install)); + } + let shrink = tool(&fx.proj, "npm", &["shrinkwrap"], &env); + if !ok(&shrink) { + soft_skip!(LEG, "`npm shrinkwrap` failed:\n{}", dump(&shrink)); + } + assert!( + fx.proj.join("npm-shrinkwrap.json").exists(), + "{LEG}: npm shrinkwrap did not produce npm-shrinkwrap.json" + ); + + let env_json = scan_hosted(&fx.proj, &[]); + assert_redirected(&env_json, "npm-shrinkwrap.json"); + assert_hosted_pin( + &read(&fx.proj.join("npm-shrinkwrap.json")), + &[NPM_UUID], + LEG, + ); +} + +#[test] +#[ignore = "live production API + real npm registry. Run with --ignored."] +fn pnpm_hosted_install_proof() { + const LEG: &str = "pnpm_hosted_install_proof"; + if !has_command("pnpm") { + soft_skip!(LEG, "`pnpm` not on PATH"); + } + let fx = npm_fixture("pnpm"); + let store = fx.cache.display().to_string(); + let env = [ + ("PNPM_HOME", store.as_str()), + ("XDG_CACHE_HOME", store.as_str()), + ]; + let store_arg = format!("--store-dir={store}"); + + let install = tool( + &fx.proj, + "pnpm", + &["install", "--ignore-scripts", &store_arg], + &env, + ); + if !ok(&install) { + soft_skip!(LEG, "upstream `pnpm install` failed:\n{}", dump(&install)); + } + // pnpm's node_modules is a symlink farm over .pnpm/; resolve through it. + let entry = fx.proj.join("node_modules").join(NPM_NAME).join("index.js"); + assert_pristine(&entry, PATCH_MARKER, LEG); + + let env_json = scan_hosted(&fx.proj, &[]); + assert_redirected(&env_json, "pnpm-lock.yaml"); + assert_hosted_pin(&read(&fx.proj.join("pnpm-lock.yaml")), &[NPM_UUID], LEG); + + std::fs::remove_dir_all(fx.proj.join("node_modules")).expect("rm node_modules"); + let reinstall = tool( + &fx.proj, + "pnpm", + &[ + "install", + "--frozen-lockfile", + "--ignore-scripts", + &store_arg, + ], + &env, + ); + + if ok(&reinstall) { + assert_patched(&entry, PATCH_MARKER, LEG); + return; + } + + // pnpm 11 added a lockfile supply-chain policy that compares every entry's + // tarball URL against the registry's published metadata. Hosted mode + // deliberately rewrites that URL to patch.socket.dev, so the policy + // rejects the lockfile: + // + // [ERR_PNPM_TARBALL_URL_MISMATCH] minimist@1.2.2 has a tarball URL + // (https://patch.socket.dev/...) that does not match the registry's + // published metadata (https://registry.npmjs.org/minimist/-/...) + // + // `--trust-lockfile` is pnpm's documented opt-out. This is a real + // compatibility gap in socket-patch's pnpm hosted mode, not a test bug: + // the CLI should emit a `redirect_pnpm_*` warning naming the flag, the way + // it already does for the gem CHECKSUMS and Rush repo-state cases. Until + // it does, this leg proves the artifact IS correctly served and installs + // cleanly once the policy is relaxed — and it fails loudly if the failure + // is anything OTHER than that known policy rejection. + let detail = dump(&reinstall); + assert!( + detail.contains("ERR_PNPM_TARBALL_URL_MISMATCH"), + "{LEG}: `pnpm install --frozen-lockfile` from the redirected lock \ + failed for an UNEXPECTED reason (not the known pnpm 11 tarball-URL \ + supply-chain policy). This is a new regression:\n{detail}" + ); + println!( + "KNOWN COMPAT GAP {LEG}: pnpm 11's lockfile supply-chain policy rejects \ + hosted-mode rewrites with ERR_PNPM_TARBALL_URL_MISMATCH. Retrying with \ + `--trust-lockfile` (pnpm's documented opt-out). socket-patch should \ + warn about this during `scan --mode hosted` on a pnpm project." + ); + + std::fs::remove_dir_all(fx.proj.join("node_modules")).ok(); + let trusted = tool( + &fx.proj, + "pnpm", + &[ + "install", + "--frozen-lockfile", + "--ignore-scripts", + "--trust-lockfile", + &store_arg, + ], + &env, + ); + assert!( + ok(&trusted), + "{LEG}: even `pnpm install --trust-lockfile` failed against the \ + redirected lock — the hosted artifact itself is not installable:\n{}", + dump(&trusted) + ); + assert_patched(&entry, PATCH_MARKER, LEG); +} + +#[test] +#[ignore = "live production API + real npm registry. Run with --ignored."] +fn yarn_classic_hosted_install_proof() { + const LEG: &str = "yarn_classic_hosted_install_proof"; + if !has_command("yarn") { + soft_skip!(LEG, "`yarn` not on PATH"); + } + let fx = npm_fixture("yarn1"); + // Without an explicit `packageManager` pin, corepack resolves a bare + // `yarn` to the latest berry (4.x) even when a classic yarn is on PATH — + // which silently turned this leg into a duplicate of the berry one. + std::fs::write( + fx.proj.join("package.json"), + format!( + r#"{{"name":"hosted-e2e","version":"0.0.0","private":true,"packageManager":"yarn@1.22.22","dependencies":{{"{NPM_NAME}":"{NPM_VERSION}"}}}}"# + ), + ) + .expect("write package.json"); + + let cache = fx.cache.display().to_string(); + let env = [ + ("YARN_CACHE_FOLDER", cache.as_str()), + ("COREPACK_ENABLE_DOWNLOAD_PROMPT", "0"), + ]; + + let version = tool(&fx.proj, "yarn", &["--version"], &env); + let major = String::from_utf8_lossy(&version.stdout).trim().to_string(); + if !ok(&version) || !major.starts_with('1') { + soft_skip!( + LEG, + "could not resolve yarn classic in this fixture (got version \ + {major:?}) — corepack may be unable to fetch yarn@1.22.22" + ); + } + + let install = tool(&fx.proj, "yarn", &["install", "--ignore-scripts"], &env); + if !ok(&install) { + soft_skip!( + LEG, + "upstream classic `yarn install` failed:\n{}", + dump(&install) + ); + } + if !fx.proj.join("yarn.lock").exists() { + soft_skip!(LEG, "`yarn install` produced no yarn.lock"); + } + assert_pristine(&minimist_entry(&fx.proj), PATCH_MARKER, LEG); + + let env_json = scan_hosted(&fx.proj, &[]); + assert_redirected(&env_json, "yarn.lock"); + assert_hosted_pin(&read(&fx.proj.join("yarn.lock")), &[NPM_UUID], LEG); + + std::fs::remove_dir_all(fx.proj.join("node_modules")).expect("rm node_modules"); + std::fs::remove_dir_all(&fx.cache).ok(); + let reinstall = tool( + &fx.proj, + "yarn", + &["install", "--frozen-lockfile", "--ignore-scripts"], + &env, + ); + assert!( + ok(&reinstall), + "{LEG}: `yarn install --frozen-lockfile` from the redirected lock \ + failed:\n{}", + dump(&reinstall) + ); + assert_patched(&minimist_entry(&fx.proj), PATCH_MARKER, LEG); +} + +#[test] +#[ignore = "live production API + real npm registry. Run with --ignored."] +fn yarn_berry_hosted_install_proof() { + const LEG: &str = "yarn_berry_hosted_install_proof"; + if !has_command("corepack") { + soft_skip!(LEG, "`corepack` not on PATH (needed to pin yarn berry)"); + } + let fx = npm_fixture("berry"); + // Berry needs an explicit packageManager pin plus the node-modules linker + // (PnP is documented as untested for hosted mode) and compressionLevel 0, + // which is what the redirect's 10c0 checksum is computed against. + std::fs::write( + fx.proj.join("package.json"), + format!( + r#"{{"name":"hosted-e2e","version":"0.0.0","private":true,"packageManager":"yarn@4.6.0","dependencies":{{"{NPM_NAME}":"{NPM_VERSION}"}}}}"# + ), + ) + .expect("write package.json"); + std::fs::write( + fx.proj.join(".yarnrc.yml"), + "nodeLinker: node-modules\ncompressionLevel: 0\nenableGlobalCache: false\n", + ) + .expect("write .yarnrc.yml"); + + let cache = fx.cache.display().to_string(); + let env = [ + ("YARN_CACHE_FOLDER", cache.as_str()), + ("YARN_GLOBAL_FOLDER", cache.as_str()), + ("COREPACK_ENABLE_DOWNLOAD_PROMPT", "0"), + ]; + + let install = tool(&fx.proj, "yarn", &["install"], &env); + if !ok(&install) { + soft_skip!( + LEG, + "berry `yarn install` failed (corepack may be unable to download \ + yarn@4.6.0):\n{}", + dump(&install) + ); + } + assert_pristine(&minimist_entry(&fx.proj), PATCH_MARKER, LEG); + + let env_json = scan_hosted(&fx.proj, &[]); + assert_redirected(&env_json, "yarn.lock"); + let lock = read(&fx.proj.join("yarn.lock")); + // Berry pins the hosted artifact through a percent-encoded `__archiveUrl` + // resolution field, so the plain host string is encoded — check both the + // encoded host and the (unencoded) patch UUID. + assert!( + lock.contains("__archiveUrl") && lock.contains("patch.socket.dev"), + "{LEG}: berry lock carries no __archiveUrl pointing at the patch \ + host:\n{lock}" + ); + assert!( + lock.contains(NPM_UUID), + "{LEG}: berry lock does not reference patch {NPM_UUID}:\n{lock}" + ); + + std::fs::remove_dir_all(fx.proj.join("node_modules")).ok(); + std::fs::remove_dir_all(fx.proj.join(".yarn")).ok(); + std::fs::remove_dir_all(&fx.cache).ok(); + let reinstall = tool(&fx.proj, "yarn", &["install", "--immutable"], &env); + assert!( + ok(&reinstall), + "{LEG}: `yarn install --immutable` from the redirected lock failed — \ + berry could not fetch the hosted artifact or its 10c0 checksum did \ + not match:\n{}", + dump(&reinstall) + ); + assert_patched(&minimist_entry(&fx.proj), PATCH_MARKER, LEG); +} + +#[test] +#[ignore = "live production API + real npm registry. Run with --ignored."] +fn bun_hosted_install_proof() { + const LEG: &str = "bun_hosted_install_proof"; + if !has_command("bun") { + soft_skip!(LEG, "`bun` not on PATH"); + } + let fx = npm_fixture("bun"); + let cache = fx.cache.display().to_string(); + let env = [("BUN_INSTALL_CACHE_DIR", cache.as_str())]; + + // Text `bun.lock` only — the binary `bun.lockb` is a separate (documented) + // auto-migration path, not what this leg covers. + let install = tool( + &fx.proj, + "bun", + &["install", "--ignore-scripts", "--save-text-lockfile"], + &env, + ); + if !ok(&install) { + soft_skip!(LEG, "upstream `bun install` failed:\n{}", dump(&install)); + } + if !fx.proj.join("bun.lock").exists() { + soft_skip!( + LEG, + "`bun install --save-text-lockfile` produced no bun.lock (bun too old?)" + ); + } + assert_pristine(&minimist_entry(&fx.proj), PATCH_MARKER, LEG); + + let env_json = scan_hosted(&fx.proj, &[]); + assert_redirected(&env_json, "bun.lock"); + assert_hosted_pin(&read(&fx.proj.join("bun.lock")), &[NPM_UUID], LEG); + + std::fs::remove_dir_all(fx.proj.join("node_modules")).expect("rm node_modules"); + std::fs::remove_dir_all(&fx.cache).ok(); + let reinstall = tool( + &fx.proj, + "bun", + &["install", "--frozen-lockfile", "--ignore-scripts"], + &env, + ); + assert!( + ok(&reinstall), + "{LEG}: `bun install --frozen-lockfile` from the redirected lock \ + failed:\n{}", + dump(&reinstall) + ); + assert_patched(&minimist_entry(&fx.proj), PATCH_MARKER, LEG); +} + +// =========================================================================== +// PyPI ecosystem — requirements.txt and uv.lock +// =========================================================================== + +/// Locate `site-packages` inside a venv, across platforms and Python minors. +fn site_packages(venv: &Path) -> Option { + if cfg!(windows) { + let p = venv.join("Lib").join("site-packages"); + return p.exists().then_some(p); + } + let lib = venv.join("lib"); + let entries = std::fs::read_dir(lib).ok()?; + for e in entries.flatten() { + let p = e.path().join("site-packages"); + if p.exists() { + return Some(p); + } + } + None +} + +/// The urllib3 patches rewrite files under the package directory; which file +/// depends on which of the three advisories the resolver picked, so look for +/// the marker anywhere in the package rather than pinning one filename. +fn urllib3_patched(site: &Path) -> bool { + let dir = site.join(PYPI_NAME); + let Ok(entries) = std::fs::read_dir(&dir) else { + return false; + }; + for e in entries.flatten() { + let is_py = e.path().extension().and_then(|s| s.to_str()) == Some("py"); + if is_py + && std::fs::read_to_string(e.path()) + .map(|b| b.contains(PATCH_MARKER)) + .unwrap_or(false) + { + return true; + } + } + false +} + +#[test] +#[ignore = "live production API + real PyPI. Run with --ignored."] +fn pypi_requirements_txt_hosted_install_proof() { + const LEG: &str = "pypi_requirements_txt_hosted_install_proof"; + if !has_command("uv") { + soft_skip!(LEG, "`uv` not on PATH"); + } + let tmp = tempfile::tempdir().expect("tempdir"); + let proj = tmp.path().join("proj"); + std::fs::create_dir_all(&proj).expect("mkdir proj"); + let uv_cache = tmp.path().join("uv-cache").display().to_string(); + let venv = proj.join(".venv"); + let venv_s = venv.display().to_string(); + let env = [ + ("UV_CACHE_DIR", uv_cache.as_str()), + ("VIRTUAL_ENV", venv_s.as_str()), + ]; + + std::fs::write( + proj.join("requirements.txt"), + format!("{PYPI_NAME}=={PYPI_VERSION}\n"), + ) + .expect("write requirements.txt"); + + if !ok(&tool(&proj, "uv", &["venv", "--quiet", ".venv"], &env)) { + soft_skip!(LEG, "`uv venv` failed"); + } + let install = tool( + &proj, + "uv", + &["pip", "install", "--quiet", "-r", "requirements.txt"], + &env, + ); + if !ok(&install) { + soft_skip!(LEG, "upstream `uv pip install` failed:\n{}", dump(&install)); + } + let Some(site) = site_packages(&venv) else { + soft_skip!(LEG, "could not locate site-packages under {venv_s}"); + }; + assert!( + !urllib3_patched(&site), + "{LEG}: the freshly-installed upstream urllib3 already carries \ + `{PATCH_MARKER}` — every downstream assertion would be vacuous" + ); + + let env_json = scan_hosted(&proj, &[]); + assert_redirected(&env_json, "requirements.txt"); + let reqs = read(&proj.join("requirements.txt")); + assert_hosted_pin(&reqs, PYPI_UUIDS, LEG); + assert!( + reqs.contains("--hash=sha256:"), + "{LEG}: rewritten requirements.txt carries no --hash pin, so pip/uv \ + would install the hosted wheel unverified:\n{reqs}" + ); + + std::fs::remove_dir_all(&venv).expect("rm venv"); + assert!( + ok(&tool(&proj, "uv", &["venv", "--quiet", ".venv"], &env)), + "{LEG}: re-creating the venv failed" + ); + let reinstall = tool( + &proj, + "uv", + &["pip", "install", "--quiet", "-r", "requirements.txt"], + &env, + ); + assert!( + ok(&reinstall), + "{LEG}: `uv pip install` from the redirected requirements.txt failed — \ + the hosted wheel could not be fetched or failed its hash check:\n{}", + dump(&reinstall) + ); + let site = site_packages(&venv).expect("site-packages after reinstall"); + assert!( + urllib3_patched(&site), + "{LEG}: reinstalled from the redirected requirements.txt, but no urllib3 \ + source file carries `{PATCH_MARKER}`" + ); +} + +#[test] +#[ignore = "live production API + real PyPI. Run with --ignored."] +fn pypi_uv_lock_hosted_install_proof() { + const LEG: &str = "pypi_uv_lock_hosted_install_proof"; + if !has_command("uv") { + soft_skip!(LEG, "`uv` not on PATH"); + } + let tmp = tempfile::tempdir().expect("tempdir"); + let proj = tmp.path().join("proj"); + std::fs::create_dir_all(&proj).expect("mkdir proj"); + let uv_cache = tmp.path().join("uv-cache").display().to_string(); + let env = [("UV_CACHE_DIR", uv_cache.as_str())]; + + std::fs::write( + proj.join("pyproject.toml"), + format!( + "[project]\nname = \"hosted-e2e\"\nversion = \"0.1.0\"\n\ + requires-python = \">=3.9\"\ndependencies = [\"{PYPI_NAME}=={PYPI_VERSION}\"]\n" + ), + ) + .expect("write pyproject.toml"); + + if !ok(&tool(&proj, "uv", &["lock", "--quiet"], &env)) { + soft_skip!(LEG, "`uv lock` failed"); + } + let sync = tool(&proj, "uv", &["sync", "--quiet"], &env); + if !ok(&sync) { + soft_skip!(LEG, "upstream `uv sync` failed:\n{}", dump(&sync)); + } + let venv = proj.join(".venv"); + let Some(site) = site_packages(&venv) else { + soft_skip!( + LEG, + "could not locate site-packages under {}", + venv.display() + ); + }; + assert!( + !urllib3_patched(&site), + "{LEG}: upstream urllib3 already carries `{PATCH_MARKER}` — vacuous" + ); + + let env_json = scan_hosted(&proj, &[]); + assert_redirected(&env_json, "uv.lock"); + let lock = read(&proj.join("uv.lock")); + assert_hosted_pin(&lock, PYPI_UUIDS, LEG); + + std::fs::remove_dir_all(&venv).expect("rm venv"); + let resync = tool(&proj, "uv", &["sync", "--frozen", "--quiet"], &env); + assert!( + ok(&resync), + "{LEG}: `uv sync --frozen` from the redirected uv.lock failed:\n{}", + dump(&resync) + ); + let site = site_packages(&venv).expect("site-packages after resync"); + assert!( + urllib3_patched(&site), + "{LEG}: resynced from the redirected uv.lock, but no urllib3 source \ + file carries `{PATCH_MARKER}`" + ); +} + +// =========================================================================== +// Cargo — per-patch sparse registry +// =========================================================================== + +#[test] +#[ignore = "live production API + real crates.io. Run with --ignored."] +fn cargo_hosted_install_proof() { + const LEG: &str = "cargo_hosted_install_proof"; + if !has_command("cargo") { + soft_skip!(LEG, "`cargo` not on PATH"); + } + let tmp = tempfile::tempdir().expect("tempdir"); + let proj = tmp.path().join("proj"); + std::fs::create_dir_all(proj.join("src")).expect("mkdir src"); + let home = tmp.path().join("cargo-home").display().to_string(); + let env = [("CARGO_HOME", home.as_str())]; + + std::fs::write( + proj.join("Cargo.toml"), + format!( + "[package]\nname = \"hosted-e2e\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n\ + [dependencies]\n{CARGO_NAME} = \"={CARGO_VERSION}\"\n" + ), + ) + .expect("write Cargo.toml"); + std::fs::write(proj.join("src").join("main.rs"), "fn main() {}\n").expect("write main.rs"); + + let fetch = tool(&proj, "cargo", &["fetch"], &env); + if !ok(&fetch) { + soft_skip!(LEG, "upstream `cargo fetch` failed:\n{}", dump(&fetch)); + } + let pristine_lock = read(&proj.join("Cargo.lock")); + assert!( + pristine_lock.contains("registry+https://github.com/rust-lang/crates.io-index"), + "{LEG}: pristine Cargo.lock does not resolve {CARGO_NAME} from \ + crates.io — fixture setup is wrong:\n{pristine_lock}" + ); + + let env_json = scan_hosted(&proj, &[]); + assert_redirected(&env_json, "Cargo.lock"); + + let lock = read(&proj.join("Cargo.lock")); + assert_hosted_pin(&lock, &[CARGO_UUID], LEG); + let config = read(&proj.join(".cargo").join("config.toml")); + assert!( + config.contains(&format!( + "sparse+https://{PATCH_HOST}/patch-registry/cargo/" + )), + "{LEG}: .cargo/config.toml declares no Socket sparse registry:\n{config}" + ); + let manifest = read(&proj.join("Cargo.toml")); + assert!( + manifest.contains(&format!("socket-patch-{CARGO_UUID}")), + "{LEG}: Cargo.toml does not route {CARGO_NAME} at the per-patch \ + registry:\n{manifest}" + ); + + // Proof: fetch again with a cold CARGO_HOME so cargo must reach the Socket + // sparse index, download the crate, and verify the checksum in the lock. + let cold = tmp.path().join("cargo-home-cold").display().to_string(); + let cold_env = [("CARGO_HOME", cold.as_str())]; + let refetch = tool(&proj, "cargo", &["fetch"], &cold_env); + assert!( + ok(&refetch), + "{LEG}: `cargo fetch` from the Socket sparse registry failed — cargo \ + could not reach the index, download the crate, or verify its \ + checksum:\n{}", + dump(&refetch) + ); + + // The extracted source must be the patched crate, not the crates.io one. + let src_root = Path::new(&cold).join("registry").join("src"); + let mut found = None; + if let Ok(hosts) = std::fs::read_dir(&src_root) { + for host in hosts.flatten() { + let candidate = host + .path() + .join(format!("{CARGO_NAME}-{CARGO_VERSION}")) + .join("src") + .join("lib.rs"); + if candidate.exists() { + found = Some(candidate); + break; + } + } + } + let lib_rs = found.unwrap_or_else(|| { + panic!("{LEG}: no extracted {CARGO_NAME}-{CARGO_VERSION}/src/lib.rs under {src_root:?}") + }); + assert!( + lib_rs + .parent() + .and_then(|p| p.parent()) + .and_then(|p| p.parent()) + .and_then(|p| p.file_name()) + .map(|n| n.to_string_lossy().contains(PATCH_HOST)) + .unwrap_or(false), + "{LEG}: {CARGO_NAME} was extracted from a non-Socket registry dir \ + ({}) — cargo served it from the crates.io cache instead of the \ + redirect", + lib_rs.display() + ); + assert_patched(&lib_rs, CARGO_MARKER, LEG); +} + +// =========================================================================== +// RubyGems — redirect works; the hosted install is blocked by a SERVER defect +// =========================================================================== + +/// The gem redirect itself is correct and is asserted hard here. +/// +/// The **install** leg is a different story. Socket's gem patch-registry serves +/// a compact index whose `/info/` line declares **no runtime +/// dependencies**, while the `.gem` it serves declares six. Bundler's +/// `ensure_same_dependencies` check fails closed: +/// +/// ```text +/// Bundler::APIResponseMismatchError: Downloading activestorage-7.0.2.2 +/// revealed dependencies not in the API (activesupport (= 7.0.2.2), ...) +/// ``` +/// +/// Compare production's own index, which does emit them: +/// `https://index.rubygems.org/info/activestorage` → +/// `7.0.2.2 actionpack:= 7.0.2.2,activejob:= 7.0.2.2,...|checksum:...` +/// versus `patch.socket.dev/patch-registry/gem///info/activestorage` +/// → `7.0.2.2 |checksum:...`. +/// +/// That is a **server-side** defect, not a CLI one, and it blocks hosted gem +/// mode for any gem with runtime dependencies. Until it is fixed the install +/// leg reports loudly but does not fail the suite; set +/// `SOCKET_PATCH_HOSTED_E2E_GEM_STRICT=1` to promote it to a hard failure +/// (do that as the regression guard once the server is fixed). +#[test] +#[ignore = "live production API + real rubygems.org. Run with --ignored."] +fn gem_bundler_hosted_redirect_and_known_install_defect() { + const LEG: &str = "gem_bundler_hosted_redirect_and_known_install_defect"; + if !has_command("ruby") || !has_command("bundle") { + soft_skip!(LEG, "`ruby` and/or `bundle` not on PATH"); + } + let tmp = tempfile::tempdir().expect("tempdir"); + let proj = tmp.path().join("proj"); + std::fs::create_dir_all(&proj).expect("mkdir proj"); + let bundle_path = tmp.path().join("bundle").display().to_string(); + let env = [ + ("BUNDLE_PATH", bundle_path.as_str()), + ("BUNDLE_APP_CONFIG", bundle_path.as_str()), + ]; + + std::fs::write( + proj.join("Gemfile"), + format!("source \"https://rubygems.org\"\ngem \"{GEM_NAME}\", \"{GEM_VERSION}\"\n"), + ) + .expect("write Gemfile"); + + // `--add-checksums` produces the CHECKSUMS section the hosted rewrite pins + // into; it needs bundler >= 2.6. + if !ok(&tool(&proj, "bundle", &["lock", "--add-checksums"], &env)) { + soft_skip!( + LEG, + "`bundle lock --add-checksums` failed (bundler < 2.6 has no \ + CHECKSUMS section)" + ); + } + let install = tool(&proj, "bundle", &["install", "--quiet"], &env); + if !ok(&install) { + soft_skip!(LEG, "upstream `bundle install` failed:\n{}", dump(&install)); + } + + let env_json = scan_hosted(&proj, &[]); + assert_redirected(&env_json, "Gemfile.lock"); + + // Hard assertions: the redirect itself must be correct. + let gemfile = read(&proj.join("Gemfile")); + assert!( + gemfile.contains(&format!("https://{PATCH_HOST}/patch-registry/gem/")) + && gemfile.contains(GEM_UUID), + "{LEG}: Gemfile carries no per-dep Socket source block for \ + {GEM_UUID}:\n{gemfile}" + ); + let lock = read(&proj.join("Gemfile.lock")); + assert!( + lock.contains("CHECKSUMS"), + "{LEG}: Gemfile.lock lost its CHECKSUMS section:\n{lock}" + ); + + // Known-broken leg: reinstall from the redirected Gemfile. + std::fs::remove_dir_all(&bundle_path).ok(); + let reinstall = tool(&proj, "bundle", &["install"], &env); + let gem_strict = std::env::var("SOCKET_PATCH_HOSTED_E2E_GEM_STRICT") + .map(|v| v == "1" || v.eq_ignore_ascii_case("true")) + .unwrap_or(false); + if ok(&reinstall) { + // The server defect has been fixed. Say so loudly — the guard below + // should be promoted to unconditional and this branch deleted. + println!( + "NOTE {LEG}: `bundle install` from the redirected Gemfile now \ + SUCCEEDS. The gem patch-registry compact-index dependency defect \ + appears to be FIXED — delete the tolerance branch in this test and \ + assert unconditionally." + ); + return; + } + let detail = dump(&reinstall); + let is_known_defect = detail.contains("APIResponseMismatchError") + || detail.contains("revealed dependencies not in the API"); + assert!( + !gem_strict, + "{LEG}: SOCKET_PATCH_HOSTED_E2E_GEM_STRICT=1 and `bundle install` from \ + the redirected Gemfile failed:\n{detail}" + ); + assert!( + is_known_defect, + "{LEG}: `bundle install` from the redirected Gemfile failed for an \ + UNEXPECTED reason (not the known compact-index dependency defect). \ + This is a new regression:\n{detail}" + ); + println!( + "KNOWN PRODUCTION DEFECT {LEG}: the Socket gem patch-registry compact \ + index omits runtime dependencies, so bundler refuses the download \ + (APIResponseMismatchError). Hosted gem mode is unusable for gems with \ + dependencies until the server emits them. Redirect assertions above \ + all passed." + ); +} + +// =========================================================================== +// Documented negative cases +// =========================================================================== + +/// Go hosted mode is refused by design (`docs/design/golang-hosted-no-go.md`). +/// +/// This asserts the *documented* shape of the refusal rather than a specific +/// warning payload, because production publishes no free golang patches today, +/// so there is nothing for the rewriter to refuse. If that ever changes, the +/// `redirect_golang_unsupported` branch below starts exercising and this test +/// becomes a real guard with no edit needed. +#[test] +#[ignore = "live production API. Run with --ignored."] +fn golang_hosted_is_refused_by_design() { + const LEG: &str = "golang_hosted_is_refused_by_design"; + if !has_command("go") { + soft_skip!(LEG, "`go` not on PATH"); + } + let tmp = tempfile::tempdir().expect("tempdir"); + let proj = tmp.path().join("proj"); + std::fs::create_dir_all(&proj).expect("mkdir proj"); + std::fs::write( + proj.join("go.mod"), + "module example.com/hosted-e2e\n\ngo 1.21\n", + ) + .expect("write go.mod"); + + let env_json = scan_hosted(&proj, &["--ecosystems", "golang"]); + assert_eq!( + redirected_count(&env_json), + 0, + "{LEG}: golang hosted mode redirected something — it is documented as \ + impossible (sumdb + module-path identity + GOPROXY leakage). Either \ + the design changed or this is a real bug:\n{env_json:#}" + ); + let warnings = env_json["redirect"]["warnings"] + .as_array() + .cloned() + .unwrap_or_default(); + if warnings + .iter() + .any(|w| w["code"].as_str() == Some("redirect_golang_unsupported")) + { + println!("{LEG}: production now publishes golang patches; the documented refusal fired."); + } else { + println!( + "{LEG}: no golang patches published, so the refusal path is inert. \ + Asserted only that hosted mode redirected nothing." + ); + } +} + +/// Deno hosted mode is not supported. Same shape as the golang guard. +#[test] +#[ignore = "live production API. Run with --ignored."] +fn deno_hosted_is_unsupported() { + const LEG: &str = "deno_hosted_is_unsupported"; + let tmp = tempfile::tempdir().expect("tempdir"); + let proj = tmp.path().join("proj"); + std::fs::create_dir_all(&proj).expect("mkdir proj"); + std::fs::write( + proj.join("deno.json"), + r#"{"imports":{"minimist":"npm:minimist@1.2.2"}}"#, + ) + .expect("write deno.json"); + + let env_json = scan_hosted(&proj, &["--ecosystems", "deno"]); + assert_eq!( + redirected_count(&env_json), + 0, + "{LEG}: deno hosted mode redirected something, but hosted mode is \ + documented as unsupported for deno:\n{env_json:#}" + ); +} + +// =========================================================================== +// Canary — ecosystems whose hosted support has nothing to test against +// =========================================================================== + +/// maven, nuget and composer all implement hosted mode, but production +/// publishes no free-tier patches for them, so there is no honest end-to-end +/// leg to write. This probes production every run and reports the moment that +/// changes, so coverage can be extended deliberately rather than by accident. +/// +/// It deliberately does NOT fail when patches appear: production publishing a +/// new patch is not a socket-patch regression, and a required check must not +/// go red for it. `SOCKET_PATCH_HOSTED_E2E_CANARY_STRICT=1` makes it fail, for +/// use in a scheduled run where a nag is the point. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "live production API. Run with --ignored."] +async fn canary_unpublished_ecosystems() { + let mut newly_published: Vec = Vec::new(); + let mut probe_errors: Vec = Vec::new(); + + for (eco, candidates) in UNPUBLISHED_ECOSYSTEMS { + for purl in *candidates { + match published_uuids(purl).await { + Ok(uuids) if !uuids.is_empty() => { + newly_published.push(format!("{eco}: {purl} -> {uuids:?}")); + } + Ok(_) => {} + Err(e) => probe_errors.push(format!("{eco}: {purl}: {e}")), + } + } + } + + assert!( + probe_errors.is_empty(), + "production probe failed (the endpoint itself may be down, which IS a \ + real signal for this suite):\n - {}", + probe_errors.join("\n - ") + ); + + if newly_published.is_empty() { + println!( + "canary_unpublished_ecosystems: maven / nuget / composer still have \ + no free-tier published patches — their hosted-mode legs remain \ + untestable end-to-end against production." + ); + return; + } + + let msg = format!( + "production now publishes free patches for previously-empty \ + ecosystems:\n - {}\nExtend this suite with real install proofs for \ + them (see docs/testing/hosted-production-e2e.md).", + newly_published.join("\n - ") + ); + if std::env::var("SOCKET_PATCH_HOSTED_E2E_CANARY_STRICT").as_deref() == Ok("1") { + panic!("{msg}"); + } + println!("NOTE canary_unpublished_ecosystems: {msg}"); +} diff --git a/docs/testing/hosted-production-e2e.md b/docs/testing/hosted-production-e2e.md new file mode 100644 index 0000000..5886e80 --- /dev/null +++ b/docs/testing/hosted-production-e2e.md @@ -0,0 +1,260 @@ +# Hosted-mode production e2e + +`crates/socket-patch-cli/tests/e2e_hosted_production.rs` is the only test suite +in this repo that exercises [hosted mode](../ecosystems.md#mode--ecosystem-matrix) +(`scan --mode hosted`) against the **real** Socket production service with **no +mocking anywhere**. Every other hosted-mode capstone (`e2e_redirect_*_build.rs`) +serves the patch artifact from a local wiremock, which proves the CLI's rewrite +grammar but cannot notice production drifting away from it. + +## What it proves + +For each ecosystem × package manager: + +1. install a pinned, known-vulnerable dependency from its **real** upstream + registry with the **real** package manager; +2. assert the installed bytes are pristine (anti-vacuity); +3. `socket-patch scan --mode hosted --json --yes` — resolves a hosted patch + reference from `patches-api.socket.dev` and rewrites the lockfile / registry + config to point at `patch.socket.dev`; +4. assert the rewrite landed (patch host + patch UUID present, integrity pin + replaced); +5. **wipe the install tree and reinstall from the rewritten lock alone** — the + package manager itself fetches from `patch.socket.dev` and verifies the + integrity pin it was handed; +6. assert the reinstalled bytes carry the patch. + +Step 5 is the point. It is the only place in this repo where a third-party +package manager — not socket-patch — downloads a Socket-hosted artifact and +independently verifies its checksum. + +## Required production patches + +The suite is pinned to these patches. They must stay **published** and +**free-tier** on `patches-api.socket.dev`; the suite runs against the +unauthenticated public proxy on purpose, because that is the surface every user +without a token gets. No API token is used, and `SOCKET_API_TOKEN` is scrubbed +from the child environment. + +| Ecosystem | PURL | Patch UUID | Advisory | Used by | +|-----------|------|------------|----------|---------| +| npm | `pkg:npm/minimist@1.2.2` | `80630680-4da6-45f9-bba8-b888e0ffd58c` | GHSA-xvch-5gv4-984h / CVE-2021-44906 | all five npm-family legs | +| PyPI | `pkg:pypi/urllib3@1.26.18` | `de58c8b8-796c-4b6d-8a48-539b5563db76`, `26242e35-f867-4da8-8789-f0d2ea49e0f1`, `e828efa5-5c6d-43f3-9909-03f5ac232b98` | GHSA-38jv-5279-wg99, GHSA-2xpw-w6gg-jr37, GHSA-gm62-xv2j-4w53 | requirements.txt, uv.lock | +| Cargo | `pkg:cargo/traitobject@0.1.1` | `cf2e6f58-d9fa-4096-9151-c34afa717f89` | GHSA-pp8r-vv2j-9j5v | cargo sparse-registry leg | +| RubyGems | `pkg:gem/activestorage@7.0.2.2` | `2535d43d-67ce-4944-be27-c19e113997fb` | GHSA-w749-p3v6-hccq | bundler leg | + +urllib3 1.26.18 carries **three** distinct free patches, one per advisory. Which +one the resolver returns is a server-side ordering detail, so the suite accepts +any of the three rather than pinning one — pinning would go red on an unrelated +server-side reorder. + +`preflight_required_patches_are_published` checks all four every run and fails +first with the offending PURL named, so a withdrawn patch produces one clear +failure instead of N confusing ones that look like CLI regressions. + +### If a required patch is withdrawn + +1. Find a replacement in the same ecosystem: + ```sh + # version-less lookup lists every patched version of a package + curl -s 'https://patches-api.socket.dev/patch/by-package/pkg%3Anpm%2Flodash' | jq + ``` + Prefer a package that is small, dependency-free, and installable by every + package manager in that ecosystem's leg. +2. Update the catalog constants at the top of `e2e_hosted_production.rs` + (`*_PURL`, `*_NAME`, `*_VERSION`, `*_UUID`) **and** the table above. +3. If the new patch does not inject the `// Socket Community Patch` header + (Cargo crates do not), pick a marker unique to the patch and set the + ecosystem's `*_MARKER` constant. + +## Ecosystem coverage, and the honest gaps + +| Ecosystem | Hosted mode | Free patches in production | Suite coverage | +|-----------|-------------|----------------------------|----------------| +| npm | ✅ | ✅ many | ✅ npm, npm-shrinkwrap, pnpm, yarn classic, yarn berry, bun | +| PyPI | ✅ (requirements.txt + uv.lock only) | ✅ many | ✅ requirements.txt, uv.lock | +| Cargo | ✅ | ✅ 1 crate | ✅ sparse registry | +| RubyGems | ✅ | ✅ 1 gem | ⚠️ redirect asserted; install blocked by a **server defect** (below) | +| Maven | ✅ | ❌ **none** | canary only | +| NuGet | ✅ | ❌ **none** | canary only | +| Composer | ✅ | ❌ **none** | canary only | +| Go | ❌ [by design](../design/golang-hosted-no-go.md) | ❌ none | negative assertion | +| Deno | ❌ not supported | — | negative assertion | + +Maven, NuGet and Composer all *implement* hosted mode, but production publishes +**zero** free-tier patches for them, so there is nothing real to redirect to. +Rather than skipping silently, `canary_unpublished_ecosystems` probes production +every run and reports the moment that changes, so coverage can be extended +deliberately. It does not fail when patches appear — production publishing a +patch is not a socket-patch regression — but +`SOCKET_PATCH_HOSTED_E2E_CANARY_STRICT=1` makes it fail, for use in a scheduled +nag run. + +PyPI's poetry / pdm / pipenv locks are **not** rewritten by hosted mode (see the +[matrix](../ecosystems.md#mode--ecosystem-matrix)); those flavors are vendored-mode +only, so there is no hosted leg to write for them. + +Two supported hosted shapes are deliberately **not** covered here: + +* **npm Rush monorepos** — hosted mode supports them (`common/config/rush/pnpm-lock.yaml` + plus per-subspace locks), but a faithful leg needs a real `rush install`, which + is a much heavier fixture than everything else in this file. It also inherits + the pnpm issue below. Covered by `e2e_redirect_rush_sim.rs` against a mock. +* **yarn berry with the PnP linker** — documented as untested for hosted mode + (the lock rewrite fires, but PnP's `.yarn/cache` resolution is not exercised). + The berry leg here pins `nodeLinker: node-modules`, matching the documented + support boundary. + +## Known issues this suite surfaced + +Both were found by running against real production, and neither is a test bug. + +### 1. `gem` — hosted mode is unusable for gems with dependencies (SERVER) + +Socket's gem patch-registry serves a compact index whose `/info/` line +declares **no runtime dependencies**, while the `.gem` it serves declares six. +Bundler's `ensure_same_dependencies` check fails closed: + +``` +Bundler::APIResponseMismatchError: Downloading activestorage-7.0.2.2 revealed +dependencies not in the API (activesupport (= 7.0.2.2), actionpack (= 7.0.2.2), +activejob (= 7.0.2.2), activerecord (= 7.0.2.2), marcel (~> 1.0), mini_mime (>= 1.1.0)). +``` + +Compare the two indexes: + +```sh +# rubygems.org — full dependency list +curl -s https://index.rubygems.org/info/activestorage | grep '^7\.0\.2\.2 ' +# 7.0.2.2 actionpack:= 7.0.2.2,activejob:= 7.0.2.2,...|checksum:7997042a... + +# Socket patch-registry — empty dependency list +curl -s "https://patch.socket.dev/patch-registry/gem///info/activestorage" +# 7.0.2.2 |checksum:89b47c6d... +``` + +**Fix belongs on the server**: the compact-index generator must emit the +gemspec's runtime dependencies. Until then the suite asserts the redirect (which +is correct) and tolerates the install failure, failing loudly if it fails for +any *other* reason. Set `SOCKET_PATCH_HOSTED_E2E_GEM_STRICT=1` to promote it to +a hard failure — do that as the regression guard once the server is fixed. + +### 2. `pnpm` — pnpm 11 rejects hosted lockfiles by default (CLI UX gap) + +pnpm 11 added a lockfile supply-chain policy that compares every entry's tarball +URL against the registry's published metadata. Hosted mode deliberately rewrites +that URL, so the policy rejects the lockfile: + +``` +[ERR_PNPM_TARBALL_URL_MISMATCH] minimist@1.2.2 has a tarball URL +(https://patch.socket.dev/...) that does not match the registry's published +metadata (https://registry.npmjs.org/minimist/-/minimist-1.2.2.tgz) +``` + +`pnpm install --trust-lockfile` is pnpm's documented opt-out and works (verified: +the patched artifact installs cleanly). Neither `--trust-policy-exclude` nor +`--no-verify-store-integrity` helps — this is a distinct check. + +**Fix belongs in the CLI**: `scan --mode hosted` should emit a `redirect_pnpm_*` +warning naming `--trust-lockfile` when it rewrites a `pnpm-lock.yaml`, the way it +already warns for `redirect_gem_no_checksums_section` and +`redirect_rush_repo_state_stale`. The suite currently retries with the flag and +reports the gap loudly. + +### 3. `uv.lock` — the `sdist` entry is rewritten to a wheel URL (CLI, minor) + +The uv.lock rewriter points the `sdist` entry at the patched **wheel** and keeps +the original sdist's `size`, producing an entry whose URL, hash and size are +mutually inconsistent: + +```toml +# pristine +sdist = { url = ".../urllib3-1.26.18.tar.gz", hash = "sha256:f8ecc1bb…", size = 305687 } +wheels = [{ url = ".../urllib3-1.26.18-py2.py3-none-any.whl", hash = "sha256:34b97092…", size = 143835 }] + +# after scan --mode hosted +sdist = { url = "…patch.socket.dev/…-py2.py3-none-any.whl", hash = "sha256:ccc9a9e0…", size = 305687 } +wheels = [{ url = "…patch.socket.dev/…-py2.py3-none-any.whl", hash = "sha256:ccc9a9e0…", size = 143835 }] +``` + +uv tolerates it today because it prefers the wheel, so the leg passes. It would +bite on a `--no-binary` resolve or a platform with no matching wheel. The +rewriter should either leave `sdist` alone or update its `size` alongside the +URL and hash. + +## Running + +```sh +# everything, soft-skipping legs whose toolchain is absent +cargo test -p socket-patch-cli --test e2e_hosted_production -- --ignored + +# one leg +cargo test -p socket-patch-cli --test e2e_hosted_production -- --ignored \ + yarn_berry_hosted_install_proof --nocapture +``` + +The suite is `#[ignore]`-gated, so it stays out of the `test` and `e2e` jobs and +runs only where it is explicitly asked for. + +### Environment knobs + +| Variable | Effect | +|----------|--------| +| `SOCKET_PATCH_HOSTED_E2E_STRICT=1` | Turn every "toolchain missing" soft-skip into a hard failure. **CI sets this** — a required check must never report green on an unexercised leg. | +| `SOCKET_PATCH_HOSTED_E2E_GEM_STRICT=1` | Promote the known gem install defect to a hard failure. | +| `SOCKET_PATCH_HOSTED_E2E_CANARY_STRICT=1` | Fail when maven/nuget/composer gain their first free published patch. | + +### Toolchains + +`npm`, `corepack` (pnpm + yarn classic + yarn berry), `bun`, `uv`, `cargo`, +`ruby` + `bundle` (**≥ 2.6** — `bundle lock --add-checksums` emits the CHECKSUMS +section the gem rewrite pins into), `go`. + +### Network egress + +`patches-api.socket.dev`, `patch.socket.dev`, `registry.npmjs.org`, `pypi.org`, +`files.pythonhosted.org`, `static.crates.io`, `index.crates.io`, `rubygems.org`. + +## CI: the `hosted-e2e` job + +Defined in `.github/workflows/ci.yml`. It is intended to be a **required** status +check in branch protection, registered under exactly the name `hosted-e2e`. + +The job deliberately has **no** job-level `if:`, **no** `needs:`, **no** matrix +and **no** `continue-on-error`. A *skipped* required check is ambiguous to branch +protection and can wedge a PR at "Expected — waiting for status", so the job +always runs and always reaches success or failure; the kill switch gates the +*steps*, not the job. + +It retries the suite up to three times with backoff, because the public proxy +intermittently returns 503 "Service temporarily over capacity" — the documented +reason the older live-API suites were pulled from the PR matrix. + +### Escape hatch — production is down and this is blocking merges + +Set a repository variable (Settings → Secrets and variables → Actions → +Variables): + +``` +HOSTED_E2E_DISABLED = true +``` + +then hit **Re-run failed jobs** on any blocked PR. `vars` is read at job-run +time, so no commit and no push is needed: the job goes green with a loud +`::warning::` and a **BYPASSED** banner in the job summary, and every open PR +clears on its next re-run. + +**Delete the variable to re-arm.** Any value other than exactly `true` (including +`yes`, `1`, `True`) leaves the suite armed — a typo must not silently disable +production coverage. + +For a single run without touching the variable: **Actions → CI → Run workflow**, +then `hosted_e2e = force` (ignore the variable) or `skip` (bypass this run). + +### Turning it on + +The job runs as soon as this lands. Making it *required* is a one-time repo +setting, done after the first green run on `main`: + +> Settings → Branches → branch protection rule for `main` → Require status +> checks to pass → add **`hosted-e2e`**. From f20110219231dad85a2224b6d7a265a605d6af19 Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Thu, 30 Jul 2026 20:52:41 -0400 Subject: [PATCH 2/4] fix(ci): unbreak the hosted-e2e job on GitHub runners Three CI-only failures, none reproducible locally: * yarn berry auto-enables hardened mode on a public-PR run, which implies `--immutable` and refuses the lockfile the FIXTURE install has to create (`YN0028`). That install now passes `--no-immutable`; the reinstall-from-redirected-lock leg keeps `--immutable`, since that one is the actual proof and must stay strict. * `pnpm` was not on PATH: a bare `corepack enable` leaves `pnpm --version` failing for a project with no `packageManager` field, which is exactly the pnpm fixture's shape. pnpm and bun now come from `npm install -g`, matching the `e2e` job above, while corepack stays for the two yarn flavors and pre-downloads both pinned versions. `COREPACK_ENABLE_DOWNLOAD_PROMPT=0` is set on both steps so the suite's own `has_command` probes can never hang. * zizmor `ref-version-mismatch`: the setup-go pin carried a `# v5` comment that did not match its SHA. Now uses the exact pin already present elsewhere in this file (v6.4.0, `cache: false`), as do all six actions this job references. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 31 ++++++++++++++----- .../tests/e2e_hosted_production.rs | 8 ++++- 2 files changed, 30 insertions(+), 9 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 95cad51..8bcc226 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -864,15 +864,28 @@ jobs: - name: Setup npm-family package managers if: steps.gate.outputs.run == 'true' - # corepack resolves the pinned pnpm / yarn-classic / yarn-berry the - # suite's fixtures request via their `packageManager` field. bun comes - # from npm for the same reason pnpm does in the `e2e` job: it is the - # one install path that works uniformly across runners. + env: + # Every corepack shim invocation — including the suite's own + # `has_command` probes — must be non-interactive, or the probe hangs + # or exits non-zero and STRICT turns that into a failed leg. + COREPACK_ENABLE_DOWNLOAD_PROMPT: '0' + # corepack resolves the yarn-classic / yarn-berry versions the suite's + # fixtures pin via `packageManager`; pre-download both so the first + # invocation inside a test isn't also a network fetch. pnpm and bun + # come from npm — the same install path the `e2e` job above uses, and + # the one that reliably puts them on PATH (a bare `corepack enable` + # leaves `pnpm --version` failing for a project with no + # `packageManager` field, which is exactly the pnpm fixture's shape). run: | set -eu corepack enable - npm install -g bun@1 - node --version && npm --version && bun --version + corepack prepare yarn@1.22.22 --activate + corepack prepare yarn@4.6.0 --activate + npm install -g pnpm@10 bun@1 + node --version + npm --version + pnpm --version + bun --version - name: Setup Python + uv if: steps.gate.outputs.run == 'true' @@ -896,9 +909,10 @@ jobs: - name: Setup Go if: steps.gate.outputs.run == 'true' - uses: actions/setup-go@d35c59abb061a4a6fb18e82ac0862c26744d6ab5 # v5 + uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0 with: - go-version: '1.23' + go-version: '1.24' + cache: false - name: Run hosted-mode production e2e if: steps.gate.outputs.run == 'true' @@ -907,6 +921,7 @@ jobs: # STRICT turns the suite's local "toolchain missing" soft-skips into # hard failures. Every toolchain it needs is installed above. SOCKET_PATCH_HOSTED_E2E_STRICT: '1' + COREPACK_ENABLE_DOWNLOAD_PROMPT: '0' run: | set -u # The public proxy intermittently returns 503 "Service temporarily diff --git a/crates/socket-patch-cli/tests/e2e_hosted_production.rs b/crates/socket-patch-cli/tests/e2e_hosted_production.rs index bda0ca9..47ce23c 100644 --- a/crates/socket-patch-cli/tests/e2e_hosted_production.rs +++ b/crates/socket-patch-cli/tests/e2e_hosted_production.rs @@ -874,7 +874,13 @@ fn yarn_berry_hosted_install_proof() { ("COREPACK_ENABLE_DOWNLOAD_PROMPT", "0"), ]; - let install = tool(&fx.proj, "yarn", &["install"], &env); + // `--no-immutable` on the FIXTURE install only. Berry auto-enables + // hardened mode on a public-PR CI run, which implies `--immutable` and + // refuses the lockfile this first install has to create (`YN0028: The + // lockfile would have been created by this install, which is explicitly + // forbidden`). The reinstall below keeps `--immutable` — that leg is the + // actual proof, and it must stay strict. + let install = tool(&fx.proj, "yarn", &["install", "--no-immutable"], &env); if !ok(&install) { soft_skip!( LEG, From 6f1a1ef4a9e6be4d335da661300832bc6011dca4 Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Thu, 30 Jul 2026 21:14:02 -0400 Subject: [PATCH 3/4] fix(ci): let corepack own pnpm in the hosted-e2e job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `corepack enable` creates the pnpm shim at the same path `npm install -g pnpm@10` writes, so the two together failed the setup step with EEXIST. pnpm now comes from `corepack prepare pnpm@10 --activate`, which also fixes the original symptom: `corepack enable` alone leaves `pnpm --version` failing for a project with no `packageManager` field — exactly the pnpm fixture's shape — because the shim has no version to resolve. `--activate` supplies that default and pre-downloads it. Only bun, which corepack does not manage, still comes from npm. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8bcc226..726464c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -869,19 +869,24 @@ jobs: # `has_command` probes — must be non-interactive, or the probe hangs # or exits non-zero and STRICT turns that into a failed leg. COREPACK_ENABLE_DOWNLOAD_PROMPT: '0' - # corepack resolves the yarn-classic / yarn-berry versions the suite's - # fixtures pin via `packageManager`; pre-download both so the first - # invocation inside a test isn't also a network fetch. pnpm and bun - # come from npm — the same install path the `e2e` job above uses, and - # the one that reliably puts them on PATH (a bare `corepack enable` - # leaves `pnpm --version` failing for a project with no - # `packageManager` field, which is exactly the pnpm fixture's shape). + # corepack owns pnpm and both yarn flavors. `corepack enable` alone is + # not enough: it installs the shims, but `pnpm --version` still fails + # for a project with no `packageManager` field — exactly the pnpm + # fixture's shape — because the shim has no version to resolve. The + # `prepare … --activate` lines set that global default AND pre-download + # each version, so the first invocation inside a test is not also a + # network fetch. + # + # pnpm must NOT come from `npm install -g` here: corepack has already + # created its shim at the same path, and npm refuses with EEXIST. Only + # bun, which corepack does not manage, comes from npm. run: | set -eu corepack enable + corepack prepare pnpm@10 --activate corepack prepare yarn@1.22.22 --activate corepack prepare yarn@4.6.0 --activate - npm install -g pnpm@10 bun@1 + npm install -g bun@1 node --version npm --version pnpm --version From a929b96af6ef4011e2f3129a8e5da4d2d6ff0bab Mon Sep 17 00:00:00 2001 From: Mikola Lysenko Date: Thu, 30 Jul 2026 21:44:24 -0400 Subject: [PATCH 4/4] fix(ci): run hosted-e2e on Node 24 so pnpm 10 works pnpm 10 imports `node:sqlite` for its store index. That module does not exist before Node 22 and is only stable in 24, so on the 20.20.2 the other jobs pin, every `pnpm install` in the suite died with ERR_UNKNOWN_BUILTIN_MODULE and STRICT turned the skip into a failure. 13 of 14 legs were already green. This job pins 24 rather than following the repo-wide 20.20.2: it drives real, current package managers against production, so it should track what users actually run. The offline suites keep their pin. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 726464c..c025d6f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -860,7 +860,13 @@ jobs: if: steps.gate.outputs.run == 'true' uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 with: - node-version: '20.20.2' + # Node 24, NOT the 20.20.2 the other jobs pin. pnpm 10 imports + # `node:sqlite` for its store index, which does not exist before + # Node 22 (and is only stable in 24) — on 20 every `pnpm install` + # dies with ERR_UNKNOWN_BUILTIN_MODULE. This suite drives real, + # current package managers against production, so it tracks what + # users actually run rather than the pin the offline suites need. + node-version: '24.x' - name: Setup npm-family package managers if: steps.gate.outputs.run == 'true'