Skip to content

#345: narrow HEAD+Range shim so DuckDB-WASM uses range requests (74 MB -> 3 MB cold load) - #348

Open
rdhyee wants to merge 3 commits into
isamplesorg:mainfrom
rdhyee:fix/345-head-range-206
Open

#345: narrow HEAD+Range shim so DuckDB-WASM uses range requests (74 MB -> 3 MB cold load)#348
rdhyee wants to merge 3 commits into
isamplesorg:mainfrom
rdhyee:fix/345-head-range-206

Conversation

@rdhyee

@rdhyee rdhyee commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Fixes the 74 MB cold load in #345 — but read the framing, because this is a deliberate standards divergence, not a bug fix.

What changes

DuckDB-WASM 1.24.0 probes for range support with exactly HEAD + Range: bytes=0- and requires a 206. We answer 200, so it logs "falling back to full HTTP read" and downloads whole files — 62.9 MB of samples_map_lite_v3.parquet when it needs ~1.5 MB.

This makes only that exact probe return 206. Every other ranged HEAD stays 200.

Why it's framed as a shim

RFC 9110 §14.2: Range is defined only for GET; servers MUST ignore it on other methods. Our 200 was correct. DuckDB's probe is the nonconforming party. So this is a compatibility measure with a named removal path — delete it when the Explorer stops using Quarto's pinned duckdb-wasm and does its own init on a conformant version.

Codex talked me out of calling this a fix, and out of the wider version I first wrote (which would have answered 206 to any ranged HEAD).

Evidence

Controlled A/B — a transparent proxy changing only this one behaviour:

Control Treatment
Cold bytes 74,202,598 3,341,812
Full-read fallbacks 8 0
Facet panel on 3G 440.6 s 94.2 s
Warm bytes (relaunch) 0 1,467,731
Warm time-to-facets 4.2 s 4.2 s

So: ~71 MB saved on first visit, ~1.47 MB cost per repeat visit, no warm time regression. The warm number is included because it could have reversed the recommendation — and the original issue omitted it.

Test coverage

test/range_contract.sh — 23 assertions against wrangler dev --local with a seeded local R2:

  • Unfixed Worker: 3 failures, all HEAD+Range
  • With this change: 23/23

Four assertions exist specifically to prove the shim doesn't widen (bytes=0-99, bytes=100-199, bytes=-100 must all stay 200). Caching, CORS, ETag-stability and ranged GET are all asserted unchanged — this Worker exists for caching, so that contract is guarded.

⚠️ How to test this WITHOUT touching production

There is no staging data host. This Worker serves data.isamples.org on a route, so a plain wrangler deploy goes straight to production for every consumer.

Safe path — deploy to a workers.dev URL instead, then point the staging Explorer at it:

cd workers/data-isamples-org

# 1. Deploy to a NON-production name, with NO route.
#    Comment out the `routes = [...]` block in wrangler.toml first,
#    or use a separate wrangler config with a different `name`.
npx wrangler deploy --name isamples-data-345-canary

# 2. Point the fork's staging Explorer at the canary (no rebuild needed —
#    the Explorer supports a data_base override):
#    https://rdhyee.github.io/isamplesorg.github.io/explorer.html?data_base=https://isamples-data-345-canary.<subdomain>.workers.dev

# 3. Verify the handshake actually changed:
curl -sI -H 'Range: bytes=0-' https://isamples-data-345-canary.<subdomain>.workers.dev/isamples_202608_h3_summary_res4.parquet
#    expect: HTTP/2 206 + content-range

# 4. Measure end-to-end (harness from #346):
python tests/playwright/bandwidth_matrix.py https://rdhyee.github.io/isamplesorg.github.io \
    --profiles unthrottled,3g-fast --budget 600 \
    --query "?data_base=https://isamples-data-345-canary.<subdomain>.workers.dev"

Only promote to the data.isamples.org route once the canary shows 206s and reduced bytes on the real edge.

What is NOT verified

Everything measured here is a local wrangler dev Worker plus a Python proxy over http://localhostnot the real Worker on the real Cloudflare edge. That canary is the gap.

Also relevant: Workers Caching appears disabled today (production returns no CF-Cache-Status, and the Worker demonstrably still sees Range). If it is ever enabled, Cloudflare strips Range, caches a full 200 and slices its own 206s — which would replace this path entirely and require re-testing.

Refs #345, #313

🤖 Generated with Claude Code

https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa

rdhyee and others added 3 commits August 6, 2026 07:16
…uests again

THE PROBLEM. A cold load of the Explorer transfers ~74 MB; it should transfer
~3 MB. DuckDB-WASM 1.24.0 (the version Quarto's OJS runtime pins) decides
whether a server supports partial reads by sending exactly
`HEAD` + `Range: bytes=0-` and requiring a 206. We answer 200, so it logs
"falling back to full HTTP read" and downloads WHOLE FILES — including
samples_map_lite_v3.parquet (62.9 MB) when it needs ~1.5 MB of it.

Measured on production: facet panel at 168s on 4G, 423s on 3G, and NEVER within
10 minutes on slow 3G.

THIS IS A WORKAROUND, NOT A BUG FIX — and the code says so. RFC 9110 §14.2 is
explicit that Range is defined only for GET and servers MUST IGNORE it on other
methods, so our 200 was CORRECT and DuckDB's probe is the nonconforming party.
Codex talked me out of framing this as a fix, and out of the wider version I
first wrote.

SCOPE is as tight as it can be:
  - ONLY the exact probe shape `bytes=0-` gets 206
  - every other ranged HEAD (bytes=0-99, bytes=100-199, bytes=-100) stays
    standards-correct at 200, so the divergence cannot leak to other clients or
    harden into an accidental contract
  - a REMOVAL PATH is named in the comment: delete this when the Explorer stops
    using Quarto's pinned duckdb-wasm and does its own init on a conformant
    version

EVIDENCE. Controlled A/B with a transparent proxy that changed only this one
behaviour: 74,202,598 B -> 3,341,812 B cold, 8 full-read fallbacks -> 0, and on
3G the facet panel 440.6s -> 94.2s (4.7x).

Warm cache measured too, since it could have reversed the recommendation: the
current whole-file path caches perfectly (0 bytes on a repeat visit) while the
ranged path re-fetches ~1.47 MB. So this trades ~71 MB on first visit for
~1.47 MB per revisit, with no warm time regression (4.2s either way).

Adds test/range_contract.sh, a 23-assertion HTTP contract test run against
`wrangler dev --local` with a seeded local R2. Against the UNFIXED Worker it
fails 3/20, all of them HEAD+Range. With this change: 23/23, including four new
assertions that specifically prove the shim does NOT widen.

NOT YET VERIFIED ON THE REAL EDGE. Everything above is a local wrangler Worker
plus a Python proxy on localhost. Deploy to a NON-PRODUCTION workers.dev URL and
canary it before touching the data.isamples.org route — see the PR body.

Refs isamplesorg#345, isamplesorg#313

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa
…samplesorg#345)

The 74 MB regression was invisible to every existing test, and was actively
MIS-CLEARED in June (ISSUE_313_FINDINGS_2026-06-26.md) by a curl probe that used
GET where DuckDB uses HEAD. This is the gate that catches that class.

Deliberately cheap: no browser, no DuckDB, a handful of bytes on the wire, 2.5s
total. Designed to run BEFORE the Playwright smoke gate so the failure is caught
before anything downloads 74 MB to discover it.

Asserts:
  - ranged GET returns 206 with a Content-Range matching the release manifest
    (passes today, always has — which is exactly why it was not sufficient alone)
  - CORS exposes Content-Range/Accept-Ranges/Content-Length, without which
    cross-origin JS cannot read them
  - DuckDB-WASM 1.24.0's HEAD+Range probe returns 206 — marked xfail until the
    isamplesorg#345 Worker shim deploys; remove the marker then
  - the isamplesorg#345 shim does NOT widen: other ranged HEADs must stay standards-correct
    at 200 with no Content-Range, so a knowing divergence cannot leak into an
    unintended contract
  - manifest sizes match what the origin actually serves (the data/doc drift class)

The HEAD test carries a long comment explaining it encodes a DELIBERATE
standards divergence (RFC 9110 section 14.2 says servers MUST ignore Range on
non-GET), and names its own deletion condition. A future reader should not
mistake it for correct HTTP.

A note on my own first draft: it reported a 404 from a partially-seeded test
origin as 'the shim has widened' — a confidently wrong diagnosis, which is the
precise failure mode this file exists to prevent. It now skips instead.

Design developed with Codex, which argued for bytes-and-protocol-shape as the
blocking signal and wall-clock timings as telemetry only.

Refs isamplesorg#345, isamplesorg#313

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa
Deploying this shim is the risky part: the production wrangler.toml binds the
Worker to the data.isamples.org/* ROUTE, and there is no staging data host — so
a plain `wrangler deploy` goes live for every consumer of that hostname
immediately, including collaborators.

The earlier instruction was 'comment out the routes block first', which is a bad
instruction: easy to forget, easy to half-revert, and a mistake is a production
incident. Replaced with a SEPARATE wrangler.canary.toml (different name, no
routes key at all) so the production config is never read or modified.

deploy-canary.sh does the whole loop and refuses to hand over a URL it has not
verified:
  - checks wrangler auth and prints the exact login command if missing
  - deploys with the canary config only
  - recovers the workers.dev URL from the deploy output
  - asserts HEAD 'bytes=0-' -> 206 with the correct Content-Range
  - asserts the shim did NOT widen: bytes=0-99 / 100-199 / -100 stay 200
  - asserts ranged and plain GET are unchanged
  - exits non-zero with 'do not promote' if any check fails
  - prints the ready-to-paste staging Explorer URL and measurement command
  - --verify re-checks an existing canary; --teardown deletes it

Validated: `wrangler deploy -c wrangler.canary.toml --dry-run` parses, lists the
R2 binding, and shows no routes; the script correctly detects the unauthenticated
state rather than failing obscurely.

Refs isamplesorg#345

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant