diff --git a/STATUS.md b/STATUS.md index 9ef6061..9f853d3 100644 --- a/STATUS.md +++ b/STATUS.md @@ -4,8 +4,8 @@ Single source of truth for "what's next." One milestone per PR/run. Autopilot: pick the one task under **NEXT**, ship it, stop. Do **not** start anything under **BLOCKED**. -_Last updated: 2026-07-03 — Milestone 6 (Web Worker WASM inference) queued; -the human approved the M4-findings candidate on 2026-07-03._ +_Last updated: 2026-07-03 — Milestone 6 (Web Worker WASM inference) done; +queue empty pending an M7 proposal._ --- @@ -128,6 +128,33 @@ the human approved the M4-findings candidate on 2026-07-03._ non-tautological (RED with production `src/` reverted, tests kept). Pre-change HEAD (rollback) `0c7a26d`. +- **Milestone 6 — Web Worker WASM inference.** On the WASM fallback the depth + pipeline now runs in a dedicated **module Web Worker** (`src/depth-worker.js`), + so the main thread keeps rendering during a pass; the **WebGPU path is + byte-identical** (it already computes off-thread). The split-by-device design + also makes the `webInitChain` poison structurally impossible: the worker only + ever inits `wasm`, the main thread only ever inits `webgpu`. Facts verified + from the installed 4.2.0 source before coding: `env.js` classifies + `DedicatedWorkerGlobalScope` as a supported web env (`IS_WEBWORKER_ENV`), and + root-exported `RawImage(data, width, height, channels)` is a plain + field-assignment constructor — so the bridge posts ImageData pixels + (buffer transferred) and the worker wraps them in a `RawImage`; the depth + comes back as a plain `{ data, width, height }` (cloned, ~200KB — we don't + own the library buffer's lifetime), exactly the subset `applyDepthToCloud` + consumes. Both call sites now hand the estimator a **canvas** (the photo + handler draws its ImageBitmap to one at native size instead of passing a + blob URL, whose in-worker reachability is unverified). `hud.timeEstimator` + wraps the bridge proxy, so the M5 readout measures the true round trip; + the fake-estimator seam (`__setEstimator`) stays main-thread and never + touches the worker — all M3–M5 tests unchanged. Worker init failure rejects + the load and clears the estimator cache (retry spawns a fresh worker); + a mid-session worker crash rejects in-flight passes (the M4 gen-token catch + tears down) and clears the cache too. Smoke test M6-A pins the photo-path + canvas input contract (RED pre-M6: it got a blob-URL string). Real-model + probe (headless → WASM → worker): a 38.2s pass rendered **558 rAF frames, + max gap 1.57s** — pre-M6 the gap was the whole pass. Pre-change HEAD + (rollback) `c61d33b`. + **Carry-over facts from M3/M4 (do not re-derive):** - Depth output contract: `depth` RawImage, Uint8, 1 channel, input W×H, 0–255 @@ -143,26 +170,8 @@ the human approved the M4-findings candidate on 2026-07-03._ ## NEXT (the one actionable task) -- **Milestone 6 — Web Worker WASM inference.** Move depth inference off the - main thread into a module Web Worker so machines on the WASM fallback keep - the 30fps orbit **during** a pass (today the main thread blocks ~4–11s per - WASM pass — M4 finding; WebGPU already computes off-thread and must keep - working, worker or not). Human approved this candidate 2026-07-03. - Constraints carried from M3–M5: - - **Verify before coding, never guess** (CLAUDE.md rule): confirm from the - installed transformers.js 4.2.0 source that the pipeline runs in a worker - and what input types survive `postMessage` (an `HTMLCanvasElement` does - NOT cross the boundary — expect `ImageBitmap`/`ImageData`/raw buffer + - `RawImage` on the worker side). If the installed source + a local runtime - probe can't settle it, that's a human checkpoint — bail, don't guess. - - Keep the probe-once device pick (the `webInitChain` poison quirk lives in - the worker now), the drop-never-queue post/consume contract, the M4 - generation-token teardown semantics, and the M5 inference-time readout - (timing must now measure the round trip the user actually experiences). - - `window.__app.__setEstimator` and the M4/M5 smoke tests must keep working — - the fake-estimator seam should stay on the main thread side of the bridge. - - ⚠️ **Effort gate (skill §1–2): this is an inference-decoupling-core task — - implement only in a session at known-high effort (`/effort high`).** +(Nothing queued — propose an M7 candidate with the human before the next +autopilot run.) --- diff --git a/src/depth-worker.js b/src/depth-worker.js new file mode 100644 index 0000000..455e926 --- /dev/null +++ b/src/depth-worker.js @@ -0,0 +1,64 @@ +// --------------------------------------------------------------------------- +// Milestone 6: dedicated module worker for WASM depth inference, so the main +// thread keeps rendering during a pass (on WASM the ONNX session blocks the +// thread that runs it — measured ~4–11s/pass in the M4 probe). +// +// transformers.js 4.2.0 runs here by design: env.js classifies +// DedicatedWorkerGlobalScope as a supported web env (IS_WEBWORKER_ENV), and +// RawImage(data, width, height, channels) is a plain field-assignment +// constructor exported from the package root — both confirmed from the +// installed source 2026-07-03. +// +// Protocol (all messages are plain objects): +// main → worker { type: 'init', model } +// worker → main { type: 'ready' } | { type: 'init-error', error } +// main → worker { type: 'infer', id, data, width, height } +// — RGBA ImageData pixels; the buffer arrives transferred. +// worker → main { type: 'result', id, data, width, height } (Uint8 depth) +// | { type: 'result', id, error } +// +// The depth result is structured-cloned, NOT transferred: its typed array +// comes out of the library's post-processing and we don't own its buffer's +// lifetime; a ~200KB copy per pass is negligible next to the pass itself. +// +// Only device:'wasm' ever initializes in this worker — the WebGPU path stays +// on the main thread (it already computes off-thread) — so the webInitChain +// poison quirk (see depth.js) cannot occur here. +// --------------------------------------------------------------------------- + +let estimator = null; +let RawImageCtor = null; + +function errMessage(err) { + return err && err.message ? String(err.message) : String(err); +} + +self.onmessage = async (e) => { + const msg = e.data; + if (msg.type === 'init') { + try { + const { pipeline, RawImage } = await import('@huggingface/transformers'); + RawImageCtor = RawImage; + estimator = await pipeline('depth-estimation', msg.model, { device: 'wasm' }); + self.postMessage({ type: 'ready' }); + } catch (err) { + self.postMessage({ type: 'init-error', error: errMessage(err) }); + } + return; + } + if (msg.type === 'infer') { + try { + const image = new RawImageCtor(msg.data, msg.width, msg.height, 4); + const { depth } = await estimator(image); + self.postMessage({ + type: 'result', + id: msg.id, + data: depth.data, + width: depth.width, + height: depth.height, + }); + } catch (err) { + self.postMessage({ type: 'result', id: msg.id, error: errMessage(err) }); + } + } +}; diff --git a/src/depth.js b/src/depth.js index 9e13757..3dd58cc 100644 --- a/src/depth.js +++ b/src/depth.js @@ -35,14 +35,22 @@ export function getSelectedDevice() { // the weight download). export function getDepthEstimator() { if (!estimatorPromise) { - estimatorPromise = loadEstimator().catch((err) => { - estimatorPromise = null; + // Cache resets are guarded by promise identity: only the load that owns + // the cache may clear it, so a test-installed fake (or a newer retry) + // never gets clobbered by a stale load's failure or worker crash. + const p = loadEstimator(() => resetCacheIf(p)).catch((err) => { + resetCacheIf(p); throw err; }); + estimatorPromise = p; } return estimatorPromise; } +function resetCacheIf(promise) { + if (estimatorPromise === promise) estimatorPromise = null; +} + // Probe WebGPU BEFORE touching transformers.js and pick the device once. // Try-webgpu-then-retry-wasm does NOT work on this library version: it chains // every web session load through a module-private promise @@ -63,11 +71,90 @@ async function pickDevice() { return 'wasm'; } -async function loadEstimator() { - const { pipeline } = await import('@huggingface/transformers'); +async function loadEstimator(resetCache) { const device = await pickDevice(); selectedDevice = device; - return pipeline('depth-estimation', MODEL, { device }); + if (device === 'webgpu') { + // WebGPU already computes off the main thread — keep the direct path. + const { pipeline } = await import('@huggingface/transformers'); + return pipeline('depth-estimation', MODEL, { device }); + } + // Milestone 6: on the WASM fallback the ONNX session blocks whatever thread + // runs it (~4–11s/pass, M4 finding) — run it in a dedicated module worker so + // orbit stays live during a pass. Splitting by device also keeps the + // webInitChain poison quirk structurally impossible: the worker only ever + // inits WASM, the main thread only ever inits WebGPU. + return createWorkerEstimator(resetCache); +} + +// Spawn the depth worker and resolve to an estimator with the same call shape +// the direct pipeline has at our call sites: async (canvas) => { depth }. +// Callers pass a canvas (both call sites in main.js do); a canvas can't cross +// postMessage, so the bridge reads it back to ImageData here (synchronous, +// bounded by the M4 512px capture cap on the hot path) and transfers the +// pixel buffer to the worker. depth comes back as a plain { data, width, +// height } — exactly the subset of the RawImage contract applyDepthToCloud +// consumes. hud.timeEstimator wraps the returned function at the call sites, +// so the M5 readout measures this full round trip. +function createWorkerEstimator(resetCache) { + return new Promise((resolve, reject) => { + const worker = new Worker(new URL('./depth-worker.js', import.meta.url), { + type: 'module', + }); + const pending = new Map(); // id → { resolve, reject } + let nextId = 0; + let dead = null; + + // A worker failure (init or crash) kills this estimator for good: reject + // everything in flight, terminate, and clear the module cache (identity- + // guarded by the caller) so the next upload spawns a fresh worker — the + // same retry semantics as a failed load. + const fail = (err) => { + dead = err; + worker.terminate(); + for (const p of pending.values()) p.reject(err); + pending.clear(); + resetCache(); + reject(err); // no-op if 'ready' already resolved us + }; + + worker.addEventListener('error', (e) => { + fail(new Error(e.message || 'depth worker crashed')); + }); + worker.addEventListener('messageerror', () => { + fail(new Error('depth worker message error')); + }); + worker.addEventListener('message', (e) => { + const msg = e.data; + if (msg.type === 'ready') { + resolve(estimatorFn); + } else if (msg.type === 'init-error') { + fail(new Error(msg.error)); + } else if (msg.type === 'result') { + const p = pending.get(msg.id); + if (!p) return; + pending.delete(msg.id); + if (msg.error) p.reject(new Error(msg.error)); + else p.resolve({ depth: { data: msg.data, width: msg.width, height: msg.height } }); + } + }); + + worker.postMessage({ type: 'init', model: MODEL }); + + function estimatorFn(canvas) { + if (dead) return Promise.reject(dead); + const ctx = canvas.getContext('2d', { willReadFrequently: true }); + const img = ctx.getImageData(0, 0, canvas.width, canvas.height); + const id = nextId++; + return new Promise((res, rej) => { + pending.set(id, { resolve: res, reject: rej }); + worker.postMessage( + { type: 'infer', id, data: img.data, width: img.width, height: img.height }, + [img.data.buffer], + ); + }); + } + }); } // Test hook (M4): swap in a fake estimator so the smoke tests can drive the diff --git a/src/main.js b/src/main.js index 5016a64..f3dc07d 100644 --- a/src/main.js +++ b/src/main.js @@ -261,12 +261,20 @@ photoInput.addEventListener('change', async () => { const estimator = await getDepthEstimator(); hud.noteDevice(getSelectedDevice()); statusEl.textContent = 'Estimating depth…'; - const url = URL.createObjectURL(file); + // M6: hand the estimator a canvas, not a blob URL — the one input type + // valid on BOTH estimator paths (the direct pipeline reads canvases, and + // the WASM worker bridge needs pixels it can post across the boundary; a + // blob URL's reachability inside a worker is exactly the kind of + // unverified assumption this repo forbids). Drawn at the photo's native + // size, so depth resolution matches the pre-M6 URL path. + const source = document.createElement('canvas'); + source.width = image.width; + source.height = image.height; + source.getContext('2d', { willReadFrequently: true }).drawImage(image, 0, 0); try { - const { depth } = await hud.timeEstimator(estimator)(url); + const { depth } = await hud.timeEstimator(estimator)(source); applyDepthToCloud(depth, image); } finally { - URL.revokeObjectURL(url); image.close(); } statusEl.textContent = `Done — ${GRID}×${GRID} points from ${file.name}`; diff --git a/tests/smoke.spec.js b/tests/smoke.spec.js index e6c067a..87c2501 100644 --- a/tests/smoke.spec.js +++ b/tests/smoke.spec.js @@ -643,3 +643,68 @@ test('milestone 4: camera-permission failure shows an error state and recovers', await expect(page.locator('#photo-input')).toBeEnabled(); expect(errors, `unexpected page errors:\n${errors.join('\n')}`).toEqual([]); }); + +// Milestone 6: WASM inference moves into a Web Worker so the main thread stays +// responsive during a pass. The worker itself can't run in CI (it loads the +// real ~100MB model — the worker-path proof is a local real-model probe, like +// M3's Z-sign), so the testable main-thread seam is the input contract that +// feeds the worker bridge: BOTH call sites must hand the estimator a CANVAS. +// A canvas converts to ImageData for postMessage; the blob-URL string the +// photo path passed before M6 does not cross the worker boundary usefully. +// The webcam path already passes a canvas (M4-A pins it); this pins the photo +// path. Fails on the pre-M6 code, which hands the estimator a string URL. +test('milestone 6: photo path feeds the estimator a canvas (worker-bridgeable input)', async ({ + page, +}) => { + const errors = []; + page.on('pageerror', (e) => errors.push(`pageerror: ${e}`)); + page.on('console', (m) => { + if (m.type() === 'error') errors.push(`console.error: ${m.text()}`); + }); + await page.goto('/'); + await page.waitForFunction(() => window.__app?.getPointCount() === 128 * 128); + + // Fake estimator that records what input type it was handed. + await page.evaluate(() => { + const s = (window.__t6 = { input: null }); + window.__app.__setEstimator((input) => { + s.input = { + isCanvas: input instanceof HTMLCanvasElement, + type: typeof input, + width: input && input.width, + height: input && input.height, + }; + return Promise.resolve({ + depth: { data: new Uint8Array(16), width: 4, height: 4 }, + }); + }); + }); + + // A real 3×2 PNG, generated in the page so no hand-rolled magic bytes. + const pngB64 = await page.evaluate(() => { + const c = document.createElement('canvas'); + c.width = 3; + c.height = 2; + const ctx = c.getContext('2d'); + ctx.fillStyle = '#f00'; + ctx.fillRect(0, 0, 3, 2); + return c.toDataURL('image/png').split(',')[1]; + }); + await page.setInputFiles('#photo-input', { + name: 'tiny.png', + mimeType: 'image/png', + buffer: Buffer.from(pngB64, 'base64'), + }); + + await expect(page.locator('#status')).toContainText(/done/i); + const rec = await page.evaluate(() => window.__t6.input); + expect(rec, 'estimator was never called').not.toBeNull(); + expect( + rec.isCanvas, + `photo path must hand the estimator a canvas, got ${rec.type} (${rec.width}×${rec.height})`, + ).toBe(true); + // Drawn from the decoded photo bitmap at its native size. + expect(rec.width).toBe(3); + expect(rec.height).toBe(2); + expect(errors, `unexpected page errors:\n${errors.join('\n')}`).toEqual([]); +});