Skip to content

Release 2026.10.24: hardening, disk cache, metrics - #1

Merged
f3l1x merged 9 commits into
masterfrom
hardening
Oct 24, 2026
Merged

f3l1x merged 9 commits into
masterfrom
hardening

Conversation

@ohmyfelix

Copy link
Copy Markdown
Contributor

Release 2026.10.24: hardening of the all-in-one image after a code and DevOps review, plus the unreleased work since 2026.10.19 (disk cache, Prometheus metrics, recovery of truncated images, removal of ?path=). Full user-facing notes are in CHANGELOG.md, which becomes the GitHub release notes.

⚠️ Breaking

  • With DOCSHOT_SIGN_KEY set, POST /upload* needs ?sign=; signed uploads reject form fields.
  • DOCSHOT_FETCH_ALLOW is anchored at the start (example\.com no longer matches https://example.com/…).
  • ?url= refuses loopback, link-local and unspecified addresses (checked on the resolved IP).
  • ?path= / DOCSHOT_ROOT are removed.
  • Settings that don't parse stop startup instead of silently using the default.

Fixes

  • Supervisor: a Gotenberg crash or restart killed docshot's own ffmpeg/bsdtar children and could steal their exit status. Only processes outside docshot's process group are touched now.
  • Shutdown: ListenAndServe returned immediately, so in-flight requests were cut off. HTTP now drains first, then Gotenberg stops.
  • Found by fuzzing: a 400-byte OLE file panics mscfb (fatal in the background cache re-check); a small .msg could allocate 4 GiB through an RTF size header.
  • Memory: upload bodies and form fields are bounded, there's a DOCSHOT_UPLOAD_MEMORY budget, and requests are streamed to Gotenberg.

CI / ops

Actions pinned to SHAs, Dependabot, an informational Trivy scan, govulncheck pinned, SBOM, provenance and a keyless cosign signature; compose cap_drop: [ALL] and no-new-privileges; DOCSHOT_LOG_FORMAT=json.

Test plan

  • make test-unit: new tests cover the supervisor (real processes), signing and auth, SSRF, upload limits, config, Gotenberg streaming and the fuzz seeds
  • make lint (gofmt, staticcheck, govulncheck, shellcheck, hadolint); actionlint on the workflow
  • make smoke, plus the hardened container (--cap-drop ALL, no-new-privileges) rendering Markdown, RTF (LibreOffice) and MP4
  • Killed Gotenberg while 4 ffmpeg processes rendered a video grid: 200, Gotenberg restarted
  • docker stop during a 4s render: the request finished with a 200, then Gotenberg stopped cleanly
  • All six fuzz targets ran clean for 20–45s after the fixes
  • Cosign signing first runs on the master push (needs id-token: write)

🤖 Generated with Claude Code

f3l1x and others added 9 commits October 23, 2026 19:47
Payloads carved from a capture or cut-off downloads now show what
survived instead of a 422:

- PNG: ignore CRCs and a missing IEND, inflate up to the first error,
  pad missing rows, fix bad filter bytes (incl. Adam7)
- progressive JPEG: keep the complete scans; baseline via ffmpeg
- GIF/WebP animations: keep the complete frames; lossy WebP and BMP
  padded to their declared size; ICO/CUR: largest icon repaired
- recover from decoder panics (gen2brain/webp on a cut animation)

Hardening found on the way:

- ffmpeg exiting 0 without a frame (truncated JPEG XL) no longer
  panics in ExitMsg
- stills decode with -xerror: a corrupted JPEG XL looped at 100% CPU
- ffmpeg/ffprobe stderr is capped at 4 KiB (was hundreds of MB)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET results are cached on disk (DOCSHOT_CACHE_DIR, default /tmp/docshot;
DOCSHOT_CACHE_SIZE LRU, default 10GB) by source + canonical options + ?v=,
so a repeated preview costs no S3 or file access; uploads are cached by
content. The index is rebuilt from the files on start, so a mounted volume
survives restarts.

- DOCSHOT_CACHE_TTL re-checks the source revision (S3 HEAD, file stat);
  stale-while-revalidate and stale-if-error, DOCSHOT_CACHE_STALE
- ETag, Last-Modified, If-Modified-Since, Cache-Status (RFC 9211),
  CDN-Cache-Control (RFC 9213), Surrogate-Key/Cache-Tag
- ?v= cache buster (immutable), DOCSHOT_CACHE_BUSTER, DELETE /cache
- hits are refused once their source is disabled or no longer allowed
- /metrics on prometheus/client_golang: per input type, sizes, pages,
  durations, cache, S3, HTTP, Go runtime and process metrics

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ADME

- Remove the ?path= source and DOCSHOT_ROOT; ?path= is now an unknown parameter (400).
- DOCSHOT_MAX_BYTES accepts units (100MB, 1GiB), default 100MB.
- Add DOCSHOT_DEFAULT_WIDTH / DOCSHOT_DEFAULT_HEIGHT for requests without a size.
- README: URL-first intro and quick start, one heading per endpoint, Storage and
  Formats sections, full env table, flow diagram (.docs/flow.svg).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docshot is a child subreaper and also runs ffmpeg, bsdtar, ddjvu… itself.
killOrphans SIGKILLed every direct child on a Gotenberg restart (in-flight
video/archive renders died with it), and the zombie reaper could Wait4 on
os/exec children, so their cmd.Wait failed with ECHILD.

Gotenberg runs in its own process group; only children outside docshot's
group are killed or reaped now, and never the running Gotenberg itself.
Tests use real processes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tenberg

- mscfb panics on a malformed 400-byte OLE header (makeslice); detect and
  msg recover and treat it as not OLE / 422.
- decompressRTF preallocated the header's raw size (up to 4 GiB) from a
  tiny .msg; capped at LZFu's 8x expansion.
- A panic while rendering becomes a JSON 500 with a logged stack instead
  of a dropped connection.
- Requests to Gotenberg are streamed (io.Pipe) instead of buffered;
  HashBytes hashes documents without a string copy (same digests).
- Fuzz targets for detect, e-mail, msg, RTF, EPUB and structured formats;
  the crasher stays as a seed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l shutdown

- With DOCSHOT_SIGN_KEY, POST /upload* needs ?sign= too (checked before the
  body is read; signed uploads reject form fields). Optional exp= expiry,
  docshot -sign … -expires 24h.
- DOCSHOT_FETCH_ALLOW is anchored at the start; ?url= dials refuse loopback,
  link-local and unspecified addresses (checked on the resolved IP).
- Uploads: MaxBytesReader, DOCSHOT_UPLOAD_MEMORY budget (1GB, then 429),
  ReadTimeout/IdleTimeout.
- Shutdown drains HTTP before stopping Gotenberg (ListenAndServe returned
  at once, cutting requests off); compose stop_grace_period 60s.
- Settings parse strictly: invalid values stop startup, naming the setting.
- Background revalidation recovers panics instead of killing the process.
- DOCSHOT_LOG_FORMAT=json; compose drops all capabilities, no-new-privileges.

BREAKING: unsigned uploads get 403 when DOCSHOT_SIGN_KEY is set; an
unanchored DOCSHOT_FETCH_ALLOW like 'example\.com' no longer matches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sign

- Actions pinned to commit SHAs; Dependabot (actions, gomod, docker) with
  a 14-day cooldown keeps them current.
- govulncheck pinned to v1.8.0 in CI and the Makefile.
- Informational Trivy scan of the built image (digest-pinned image).
- Published images carry an SBOM, max provenance and a keyless cosign
  signature.
- AGENTS.md: process-group rule, shutdown order, fuzzing, strict settings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add CHANGELOG.md (user-facing: Breaking / Added / Fixed / Changed per
version); CI uses the version's section as the GitHub release notes,
followed by GitHub's generated list of changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
setup-go installs the exact go.mod version, so CI still ran 1.27.2 and
govulncheck failed; the image build already used 1.27.3.
@f3l1x
f3l1x merged commit 1f2e7f6 into master Oct 24, 2026
5 checks passed
@f3l1x
f3l1x deleted the hardening branch October 24, 2026 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants