jpegio: a CircuitPython-compatible JPEG decoder as a native module, one TJpgDec per firmware

[bdbarnett]

Brad, 2026-09-03: "I like your idea" — usbif is wiring a UVC camera through board_config.display_drv and today the only JPEG decoder in the workspace is LVGL's, so the camera path needs LVGL in the build. This issue is the plan; pydevices/docs/jpegio-vision.md is the one-page version.

What. Port CircuitPython's jpegio (shared-module/jpegio/JpegDecoder.c, TJpgDec R0.03 underneath) as a native module in displayif: JpegDecoder.open(buffer|file), decode(bitmap, scale=…), same API as CP so CP builds use the native one and MicroPython builds get ours. Camera path becomes MJPEG frame → jpegio → display_drv.blit_rect, no LVGL.

One decoder per firmware. jpegio vendors tjpgd.c; its micropython.cmake/.mk compile it only when the lvgl usermod is NOT in the build, and link against LVGL's jd_prepare/jd_decomp when it is — same pinned R0.03 either way. The org-wide TJpgDec config is CircuitPython's (JD_FORMAT 1 RGB565, JD_USE_SCALE 1); LVGL's decoder is made to honour it (lvgl-bindings issue, linked below), which also un-disables TJPGD on the CircuitPython bindings.

Contract, from the usbif session's measurements (Logitech C920e, 320x240, one camera):

• UVC MJPEG frames are NOT JFIF-first: SOI → DQT → SOF0 → DHT×4 → COM → APP0/JFIF → DRI → SOS. LVGL's is_jpg() (lv_tjpgd.c:302) demands FF D8 FF E0 … "JFIF" in bytes 0..9 and rejects every webcam frame. jpegio must not sniff beyond SOI; hand the buffer to jd_prepare and let it judge (CP's behaviour).
• This camera sends its DHT tables; many UVC cameras omit them and expect the host to inject the standard ones. Decide explicitly: "valid JPEG in" (something upstream injects tables) or jpegio injects. "Cameras always send DHT" is unproven with n=1.
• Frames carry DRI (restart intervals). The test frame for any config change must have restart markers, not a clean still.
• Output: native-order RGB565 (the driver byteswaps itself). Two things worth more than raw speed: block/row-wise output through TJpgDec's output callback so a preview blits incrementally without a full-frame buffer (600 KB not allocated at 640x480), and exposing the 1/2, 1/4, 1/8 scale factor (pydevices has no resampler; the usbif example upscaled by hand).
• S3 headroom (ESP32_GENERIC_S3, 8 MB PSRAM, 800x480 panel live): ~6.5 MB contiguous free; TJpgDec's ~3 KB work buffer is nothing. Caveat: pooled PSRAM+SRAM number, internal-SRAM headroom not separated, PSRAM decode speed unmeasured.
• Target: on the S3's full-speed host the camera negotiates 320x240 @ 5 fps (~8 KB/frame) — ~5 decodes/s, undemanding. The P4's high-speed host (unproven bench) and its hardware JPEG decoder are where rate matters: keep the decode backend swappable.
• No decode rate exists yet (no successful decode on the bench); do not quote one.

Method note worth keeping: the usbif session's first LVGL attempt "worked" only because the header carried its own w:320,h:240 and LVGL echoed them back; zeroing the header exposed that nothing decoded. Any jpegio check starts from a zeroed header.

Order: decoder module + tests (a DRI frame, a table-less frame, scale factors, block callback) → cmods one-copy build rule → usbif consumer → lvgl-bindings JD_FORMAT change and CP TJPGD re-enable. Backend swap for the P4 later.

[bdbarnett]

Corpus notes from the usbif session (2026-09-03): one real frame is in the corpus — c920e_320x240_dri.jpg, 8,064 bytes, Logitech C920e (046d:08b6) hosted by an ESP32-S3 full-speed port, MJPEG format 2 / frame 6 (320x240) at 5 fps (384 B/bus-frame, alt 2); four DHT tables present; DRI present; EOI at offset 8040 with 22 bytes of trailing padding kept on purpose — it is what the USB layer hands a decoder, so "bytes after EOI must not break decode" is a test, not a cleanup. The 160x120 frames were never dumped (only their sizes: 1184–1216 B during auto-exposure settling, then ~3552 B); corpus item for the next bench-up: capture one early-exposure 160x120 frame, because a sparse settling frame is a fair thing to test a decoder against. Not blocking Phase 1.

[bdbarnett]

Phase 1 landed (2026-09-03), six commits on main. src/jpegio/ (CP's JpegDecoder API on TJpgDec R0.03 with CP's config; buffer or per-block callback targets; SOI-only sniff; JRESULT→exception table) and tests/jpegio/ (a 9-frame corpus incl. the real C920e frame, a Pillow fidelity reference, 28 TJpgDec golden digests, test_jpegio.py run by clean-build.yml).

Gate, as met: every corpus frame at scales 0–3 digested on cmods/bin/micropython built beside LVGL (lvgl.tjpgd_init and jpegio both present); callback path covers every pixel exactly once and allocates nothing per block; DHT-less frame refused at open() with JDR_FMT1 — the "who injects tables" decision is still open, and it is this issue's; progressive refused with JDR_FMT3; truncated/bare-SOI/non-JPEG/non-source inputs raise, never crash. Review: 4,620-run byte-mutation fuzz clean; planted no-op-decode and byteswap faults both caught by the tests; one real out-of-bounds write (size_t overflow in the buffer-fit check, worse on 32-bit targets) and one segfault (open() on a non-source object) found and fixed before landing.

Interim you should know about: in a firmware that also carries LVGL, both TJpgDec copies link — jpegio's two entry points are prefixed (jpegio_jd_*) so the ~6 KB duplicate is tolerated until Phase 2 unifies the config and drops to one copy (lvgl-bindings#14 carries that half).

Corpus facts that corrected the plan: the C920e's DRI interval is 0 (no restart markers in the scan; the synthetic DRI frames are what exercise them), its APP0 is not JFIF (14 zero bytes — an is_jpg-style sniff fails on identity, not just order), and 22 bytes trail its EOI (kept; decode must not choke on them, and doesn't). Still owed to the corpus: a 160x120 early-exposure frame (usbif session, next bench-up).

[bdbarnett]

Audit 2026-09-04: the module is delivered; three asks remain.

Done: src/jpegio/ and tests/jpegio/ exist on main at 42fcf81 (def186f..42fcf81), and the one-copy rule is real rather than asserted — micropython.cmake sets JPEGIO_LVGL by detecting the sibling, and a build beside LVGL has exactly one object defining jd_prepare.

Remaining:

1. The DHT-injection decision, which this issue explicitly owns. src/jpegio/README.md:143-146 still says it is an open decision recorded in the vision doc, and the vision doc points back here.
2. Phase 3, the usbif consumer. grep -rn jpegio usbif/ returns one hit, a comment in examples/uvc_display.py:24 saying the module "is being added". No import, no MJPEG branch, and no decodes-per-second figure — the number the vision doc says this owes.
3. The Break criterion (a table-less MJPEG frame from a second camera) and the owed corpus item (one early-exposure 160x120 frame).

The P4 backend swap is deferred by the issue itself, so it is not a gap.