Repository navigation
perf: retro-optimize committed markdown PNGs; decide ellipse/modeling render #247
Description
Overview
Two deferred items from the batch-2a markdown-renderings rollout (autolens_workspace#264). Investigation before planning corrected the prompt's framing on both:
PNG size. optimize_pngs() already exists in PyAutoHands/autohands/generate_markdown.py:240 and runs after every extraction (:379) — it is forward-only by construction and never touches previously committed images. So the leftover is not "add an optimize step"; it is a one-off retro pass over already-committed PNGs. pngquant is the wrong tool: reusing optimize_pngs keeps retro-optimized files byte-identical to what a future re-render produces.
Measured across the six workspaces: 405 of 429 committed markdown PNGs are still RGBA (unoptimized); the 24 P-mode ones were rendered after PR #200. Running a 25-file sample through the existing algorithm: 88.4 MB → ~23 MB (26.1%) — roughly 65 MB saved, no quality loss beyond the 256-colour quantize already shipping.
| Repo | PNGs | RGBA (to fix) | markdown/ |
|---|---|---|---|
| autolens_workspace | 207 | 196 | 61 M |
| autogalaxy_workspace | 122 | 115 | 22 M |
| HowToGalaxy | 57 | 57 | 3.9 M |
| HowToFit | 31 | 31 | 1.5 M |
| autofit_workspace | 6 | 6 | 568 K |
| HowToLens | 94 | 0 | 3.1 M (already done) |
Caveat on the record: this shrinks the working tree and future clone-of-tip, not .git history — the old blobs stay in the pack. No history rewrite is on the table.
ellipse/modeling. autogalaxy_workspace/scripts/ellipse/modeling.py runs 22 sequential fits (2 x 10 DynestyStatic + 2 Drawer, lines 445-583), split across two notebook cells by the """ section markers — so it is two multi-hour cells, not one. There is currently no output/ellipse* in the workspace, so a render starts from zero. Rather than gamble on a bumped per-cell timeout, pre-run the script outside nbconvert so the fits land in output/, then let the render resume them near-instantly (the module docstring guarantees PyAutoFit completed-run resume). The max_minutes bump becomes a safety margin rather than a bet.
Plan
- Add a
--optimize-onlymode togenerate_markdown.pythat reuses the existingoptimize_pngs()over already-committedmarkdown/**/*_files/, and run it across all six workspaces — one deterministic ~65 MB win. - No dpi knob: quantize already delivers 74%, and a dpi drop would degrade every future render's readability for a smaller marginal gain.
- For
ellipse/modeling.py, measure one fit first; if the extrapolated 22-fit cost is under a 2-hour ceiling, pre-run the script to populateoutput/, then render via the resume path. Above the ceiling, leave the page out (as the prompt permits) and replace the yaml's "tracked as a follow-up" comment with a settled decision recording the measured cost. - Ship library-first (PyAutoHands), then the workspace commits.
Detailed implementation plan
Work Classification
Library (PyAutoHands tool change first; workspace commits follow).
Affected Repositories
- PyAutoLabs/PyAutoHands (primary)
- PyAutoLabs/autolens_workspace
- PyAutoLabs/autogalaxy_workspace
- PyAutoLabs/autofit_workspace
- PyAutoLabs/HowToGalaxy
- PyAutoLabs/HowToFit
- PyAutoLabs/HowToLens (expected near-no-op)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| ./PyAutoHands | main | clean |
| ./autogalaxy_workspace | main | clean |
| ./autolens_workspace | main | clean |
| ./autofit_workspace | main | clean |
| ./HowToGalaxy | main | clean |
| ./HowToFit | main | clean |
| ./HowToLens | main | clean |
worktree_list_claimed empty; worktree_check_conflict exit 0 (no conflict). Unregistered-work warning: autogalaxy_workspace has a local feature/stored-sample-reconstruction-guard branch, not checked out and not claimed — not a conflict.
Suggested branch: feature/markdown-renderings-2a-leftovers
Worktree root: ~/Code/PyAutoLabs-wt/markdown-renderings-2a-leftovers/
Implementation Steps
Phase A — PyAutoHands (library)
autohands/generate_markdown.py: add--optimize-onlyto the argparse block (:439). When set, skipload_examples/ render /write_index; walkPath.cwd()/markdownfor*_files/dirs, call the existingoptimize_pngs(files_dir)on each, print a per-repo before/after byte total. ~20 lines, no new dependency (Pillow already imported insideoptimize_pngs).- Do not add a dpi knob.
- Update the module docstring and
docs/internals.md(if it documents this tool) to note the retro mode.
Phase B — PNG retro pass (workspaces, mechanical)
- Per repo in
autolens_workspace, autogalaxy_workspace, autofit_workspace, HowToGalaxy, HowToFit, HowToLens, from the workspace root:python ../PyAutoHands/autohands/generate_markdown.py <project> --optimize-only. - Verify: re-run the PNG mode census (expect all
P), spot-check 3 rendered pages per repo, and confirmgit statusshows onlymarkdown/**/*.pngmodified — nothing outsidemarkdown/. - HowToLens is expected to be a near-no-op; if it produces zero changes, no commit there.
Phase C — ellipse/modeling (measure, then decide)
- From
autogalaxy_workspace, time a single fit in isolation (fit_0of the Multiple-Ellipses loop,n_live=50, 2 free params) and extrapolate x22. Note this machine is WSL with a throttlednproc, so the script'snumber_of_cores=4may itself be the cost driver — check that before blaming the sampler. Ceiling: 2 hours for the full 22-fit pre-run. - If under the ceiling: run
python scripts/ellipse/modeling.pydirectly to populateoutput/, thengit statusand restore any tracked file the script mutated — running outsidegenerate_markdown.pyloses its tracked-file protection, anddataset/imaging/ellipsedoes not currently exist so the auto-simulator will fire. Then add the entry toconfig/build/markdown_examples.yamlwithmax_minutes: 360, rungenerate_markdown.py autogalaxy --only ellipse/modeling(resume path), and replace the exclusion comment. - If over the ceiling: leave the page out, and rewrite the yaml comment from "Tracked as a follow-up" to a closed decision recording the measured cost — so this does not come back a third time.
Phase D — ship
ship_libraryfor PyAutoHands, thenship_workspacefor the workspace commits behind the library-first merge gate.ship_workspacehas a known binary-leak trap — this task is all binaries by design, so state explicitly thatmarkdown/**/*.pngis the intended payload and check nothing else rides along.
Key Files
PyAutoHands/autohands/generate_markdown.py—optimize_pngs()at:240, called at:379;main()/argparse at:439autogalaxy_workspace/config/build/markdown_examples.yaml— curated list; ellipse/modeling exclusion commentautogalaxy_workspace/scripts/ellipse/modeling.py— 22-fit Multiple Ellipses + Masking sections, lines 445-583<workspace>/markdown/**/*_files/*.png— the 405 RGBA images to retro-optimize
Testing approach
No unit tests exist for generate_markdown.py. Verification is empirical: the PNG mode census before/after, byte totals per repo, git status scoped to markdown/, and visual spot-checks of rendered pages on the PR.
Original Prompt
Click to expand starting prompt
# Markdown renderings batch 2a — leftovers (ellipse/modeling + PNG size)
Type: docs
Target: workspaces
Difficulty: small
Autonomy: safe
Priority: low
Status: formalised
Two deferred items from batch 2a ([[markdown-example-renderings]] rollout,
autolens_workspace#264):
1. **autogalaxy `ellipse/modeling.py`** — excluded from the curated set because
its "Multiple Ellipses" section runs many sequential DynestyStatic/Drawer
fits that exceed nbconvert's per-cell timeout (CellTimeoutError at 7200s;
confirmed a timeout, NOT a bug — corner works, imaging/modeling renders
fine). To add it: bump its `max_minutes` in
`autogalaxy_workspace/config/build/markdown_examples.yaml` to something
generous (try 360+; the multi-fit cell is genuinely multi-hour) and run
`generate_markdown.py autogalaxy --only ellipse/modeling`. If it's still
impractical, leave it out — ellipse/simulator + ellipse/fit already showcase
the dataset and a fit.
2. **PNG size** — batch 2a committed large image galleries (autolens markdown/
~61M total incl. phase-1's 19M; autogalaxy ~22M) at generate_markdown's
default dpi. If repo size becomes a concern, do a single optimization pass
across ALL markdown/*_files/*.png in every workspace (pngquant lossy ~60-70%
reduction, visually fine for plots) — or add a dpi/optimize step to
generate_markdown.py so it's consistent going forward (would also cover the
phase-1 autolens pages). Do it repo-wide for consistency, not piecemeal.
Activity
Jammy2211 commentedon Aug 20, 2026
Correction to the detailed plan — testing approach.
The plan's "Testing approach" says "No unit tests exist for
generate_markdown.py". That is wrong:tests/test_generate_markdown.pyexists (317 lines) and already contains aTestOptimizePngsclass coveringoptimize_pngs()in-place shrink and the missing-dir no-op.Phase A therefore gains a step: extend that file with a
TestOptimizeOnlyclass covering the new--optimize-onlywalk — atmp_pathworkspace with a couple ofmarkdown/<page>_files/dirs, asserting every PNG shrinks, that nothing outsidemarkdown/is touched, and that an absentmarkdown/dir is a clean no-op. Run withpytest tests/test_generate_markdown.py.The empirical checks in the original plan (PNG mode census, per-repo byte totals,
git statusscoped tomarkdown/, visual spot-checks) still stand for Phase B — they verify the corpus, the unit test verifies the tool.Jammy2211 commentedon Aug 20, 2026
Library PR Created
PyAutoHands#248 —
feat(generate_markdown): --optimize-only retro pass over committed PNGs(pending-release, MERGEABLE).API Changes
None.
--optimize-onlyis a new opt-in flag; default behaviour is byte-for-byte unchanged.optimize_pngs()now returns(before, after)instead ofNone— internal helper, one in-tree caller, no downstream consumer.Workspace impact — option (i), new content, workspace PRs follow
The workspace repos are not affected by an API change; they consume the new flag's output. Six workspaces were run through it, and the results are committed on the same branch in each repo:
HowToLens reclaimed exactly zero — it was rendered post-#200 and is already fully optimized. No commit there; its branch carries nothing.
Phase C — the exclusion premise did not reproduce
The prompt recorded
ellipse/modeling.pyas "genuinely multi-hour" after a 7200sCellTimeoutError. Measured before acting on it: oneDynestyStaticfit takes ~32s, flat inmajor_axis(0.3" → 32.1s, 3.6" → 32.7s), so there is no blow-up at the large end that diagnosis implied.Post-#246 the notebook splits the work into two heavy cells (10 and 11 fits), so the per-cell timeout has to cover ~11 fits ≈ 6 min, not all 23. The full page rendered in 591s (9.9 min) end to end, covering Multiple Ellipses, Final Fit and Masking. Added at
max_minutes: 120(~20× headroom), with the measurement recorded in the yaml comment so the "multi-hour" claim is not re-derived from the old note.Incidental fix: regenerating the index repaired three pre-existing broken links — the committed
markdown/README.mdpointed atmarkdown/multi/{start_here,simulator,modeling}.md, but those pages live undermarkdown/multi_dataset/. Verified the old targets do not exist and the new ones do.Gate
pytest tests/in PyAutoHands: 354 passed (5 new).stale(85), sole reasonrelease validation stale: source moved since rehearsal (PyAutoGalaxy)— unrelated to this work. No RED, no YELLOW. Nothing merged; merge is held for a human.Workspace PRs next via
/ship_workspace.Jammy2211 commentedon Aug 20, 2026
Workspace PRs Created — all six PRs open, nothing merged
--optimize-onlyflag + 5 testsellipse/modelingpage + index fix, 21.5 → 5.6 MBAll five workspace PRs carry
pending-release; all six areMERGEABLE.HowToLens has an empty branch and no PR — it was rendered post-#200 and reclaimed exactly 0 bytes, as predicted.
Merge order (library-first gate)
PyAutoHands#248 must merge first — it adds the flag the workspace contents were produced with. The five workspace PRs follow in any order.
Not done
active.mdentry is not pruned — both wait on merge.stale(85), sole reasonrelease validation stale: source moved since rehearsal (PyAutoGalaxy). Not GREEN, but the reason is PyAutoGalaxy release-rehearsal drift and unrelated to this work; no RED, no YELLOW. Flagging it rather than treating it as a pass..githistory is unchanged — the 88 MB of original blobs remain in every pack. The saving is to the working tree and future clone-of-tip only.Jammy2211 commentedon Aug 20, 2026
Shipped — all six PRs merged 2026-08-20
cdea28cd45cdca5c91a0e51c3df684e1e6b48e2ad02Merged library-first (#248 before the workspace PRs). All 35 CI legs were green beforehand.
Verified on
mainafter mergePP(122 + 7 new ellipse)PPPPBoth prompt premises were wrong
optimize_pngs()already existed (feat: markdown generator per-cell stdout truncation + PNG optimization #200); being forward-only was the actual gap. Fixed with a retro pass reusing the same function, not the suggested pngquant sweep, so retro bytes match what a future re-render emits.major_axis(0.3″ → 32.1 s, 3.6″ → 32.7 s); the whole page rendered in 591 s. Added atmax_minutes: 120with the measurement recorded in the yaml comment so the false claim can't be re-derived.Incidental: three pre-existing broken index links repaired (
markdown/multi/…→markdown/multi_dataset/…).Standing caveat
.githistory is unchanged — the ~88 MB of original blobs remain in every pack. The saving is to the working tree and future clone-of-tip only.Completion record:
PyAutoMind/complete/2026/08/markdown-renderings-2a-leftovers.md.