Repository navigation
fix: audit and repair broken doc URLs across PyAuto repos #508
Description
Overview
Documentation URLs across the PyAuto repos have rotted. The Google Colab links in the PyAutoLens new-user guide include a literal hhttps:// typo; dozens of Jammy2211/<workspace> GitHub paths now 301-redirect to PyAutoLabs/<workspace>; and they reference a release branch that no longer exists on either owner. There is also no automated check to catch this rot before users hit it. This task adds a reusable URL-audit tool to admin_jammy/software/ and uses it to clean up every broken doc URL across the 12 PyAuto repos.
Plan
- Build a Python URL-checker in
admin_jammy/software/url_check/that scans repo paths, dedupes URLs, and HEAD/GET-validates them in parallel. Forcolab.research.google.com/github/<owner>/<repo>/blob/<ref>/<path>links, validate the underlyingraw.githubusercontent.com/<owner>/<repo>/<ref>/<path>instead — Colab returns HTTP 200 even for dead refs, so the colab URL itself can't be trusted. - Run the audit across all PyAuto repos and produce a structured Markdown + JSON report listing each broken URL with
repo:file:linelocations. - Apply scripted fixes for the three known broken patterns:
hhttps://→https://;Jammy2211/{autolens,autogalaxy,autofit}_workspace→PyAutoLabs/...;/blob/release/→/blob/main/. - Investigate and fix any remaining broken URLs the audit flags (arxiv DOIs, readthedocs paths, dead external refs).
- Ship per-repo PRs. Doc-only changes have no API impact, so library-first gating doesn't apply.
Detailed implementation plan
Affected Repositories
- PyAutoLens (primary — where the bug was reported)
- admin_jammy (tool lives here)
- PyAutoConf, PyAutoFit, PyAutoArray, PyAutoGalaxy
- HowToFit, HowToGalaxy, HowToLens
- autofit_workspace, autogalaxy_workspace, autolens_workspace
Work Classification
Both (library docs + workspace tutorial scripts). Library-first gate not required: doc-only, no API surface.
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| admin_jammy | main | 2 unrelated (papers.md, week.md) |
| PyAutoConf | main | clean |
| PyAutoFit | main | clean (feature/priors-jax-native lives in another worktree) |
| PyAutoArray | main | clean |
| PyAutoGalaxy | main | clean (feature/fit-ellipse-jax lives in another worktree) |
| PyAutoLens | main | clean |
| HowToFit | main | clean |
| HowToGalaxy | main | clean |
| HowToLens | main | 2 regenerated dataset files (ignored) |
| autofit_workspace | main | clean |
| autogalaxy_workspace | main | 8 regenerated dataset files (ignored) |
| autolens_workspace | main | 17 regenerated dataset files (ignored) |
Conflict status: worktree_check_conflict flags PyAutoFit and PyAutoGalaxy as claimed by other tasks, but those tasks edit source code and this task only touches docs/ and tutorial scripts — explicitly cleared by the user to proceed in parallel.
Suggested branch: feature/url-check
Worktree root: ~/Code/PyAutoLabs-wt/url-check/ (created later by /start_library)
Implementation Steps
-
Build the audit tool —
admin_jammy/software/url_check/url_check.py:- Regex-extract URLs from
*.md,*.rst,*.py,*.ipynb,*.txt,*.yml,README*(strip trailing).,;!) - Dedupe; keep
{url: [(repo, file, line), ...]}index - Parallel HEAD requests (~16-thread pool); fall back to GET on 405/403
- Colab handler: rewrite
colab.research.google.com/github/<owner>/<repo>/blob/<ref>/<path>→raw.githubusercontent.com/<owner>/<repo>/<ref>/<path>and HEAD-check that - Categories:
ok(2xx),redirect(3xx — record final URL),broken(4xx/5xx/timeout/DNS) - Output:
admin_jammy/software/url_check/report.md+report.json --fix-known-patternsflag runs the scripted rewrites listed belowrun.shwrapper invokes the audit against all PyAuto repos by default
- Regex-extract URLs from
-
Scripted fixes for the known patterns:
hhttps://→https://(2 known sites:PyAutoLens/docs/overview/overview_2_new_user_guide.md:31,PyAutoGalaxy/docs/overview/overview_2_new_user_guide.md:17)github.com/Jammy2211/{autolens,autogalaxy,autofit}_workspace→github.com/PyAutoLabs/...(~113 sites)/blob/release/notebooks/→/blob/main/notebooks/- Verify each rewrite returns 200 before committing
-
Case-by-case fixes for whatever the audit flags beyond known patterns (arxiv, readthedocs, ADS, JOSS, missing PNG image URLs from
raw.githubusercontent.com/...). -
Colab verification pass: confirm every Colab URL's underlying notebook exists. Tag-pinned URLs (
blob/<version-tag>/...) are left as-is if the tag resolves; flagged if not. -
Ship per repo as
feature/url-checkPRs via/ship_libraryor/ship_workspace.
Key Files
admin_jammy/software/url_check/url_check.py— new audit tooladmin_jammy/software/url_check/run.sh— wrapperadmin_jammy/software/url_check/report.md+report.json— generated artifactsPyAutoLens/docs/overview/overview_2_new_user_guide.md— typo + pattern fixesPyAutoGalaxy/docs/overview/overview_2_new_user_guide.md— same- Other
docs/*.{md,rst}across libraries — pattern rewrites - Workspace
*.pytutorial scripts — pattern rewrites - README files across repos — pattern rewrites
Testing
- Re-run the audit after fixes; expect zero
broken, only acceptableredirects. - Manually open 3-5 fixed Colab links to confirm load.
- No code/test changes needed (doc-only).
Original Prompt
Click to expand starting prompt
URL links to Google colabs are not correct, linking to broken colabs which dont run, for example
in the PyAutoLens docs:
https://pyautolens.readthedocs.io/en/latest/overview/overview_2_new_user_guide.html
Here:
CDD Imaging: For image data from telescopes like Hubble and James Webb, go to imaging/start_here.ipynb.
Can you scan all repos for all URLs and check they work, in particular making sure the google colab pages
that come up actrually run?
Activity
Jammy2211 commentedon May 15, 2026
Library phase shipped — 6 PRs open across the libraries + tooling.
The audit tool now lives at
admin_jammy/software/url_check/and is reusable. We scanned all 12 PyAuto repos (~3000 unique URLs, ~125k locations), then applied ~20 classes of scripted rewrites driven by the patterns the audit surfaced.What the audit found before fixes: 304 broken URLs, including the
hhttps://typo you spotted plus a long tail of staleJammy2211/<workspace>paths, references to areleasebranch that no longer exists on the workspaces, thejoshspeagle/nautilussampler having moved tojohannesulf/nautilus, bokeh's CoC moving intodocs/, sphinx-doc renaming/en/main→/en/master, and a chunk of workspace notebooks that were reorganised (e.g.modeling/imaging/features/<x>.ipynbis nowimaging/features/<x>/modeling.ipynb).What's fixed in this wave (library-side):
Your reported bug — the
hhttps://typo on overview_2_new_user_guide line 31 — is in #509 along with the same typo in PyAutoGalaxy. All library unit tests pass (3236 tests total across the 5 libraries). Line endings are preserved, so the diffs are pure URL changes.Still to come (workspace phase): the audit also flagged URLs inside the workspace repos themselves (HowToFit, HowToGalaxy, HowToLens, autofit_workspace, autogalaxy_workspace, autolens_workspace). Those weren't touched in this wave — they'll get the same rewrites in a workspace-side PR after the library PRs merge. After both waves, broken count should drop from 304 → ~50 (the remainder are external paywalled / dead links we can't auto-fix).
docs: audit-driven URL fixes (Jammy2211 → PyAutoLabs, /release/ → /ma…
docs: audit-driven URL fixes (Jammy2211 → PyAutoLabs, /release/ → /ma…
docs: audit-driven URL fixes (Jammy2211 → PyAutoLabs, /release/ → /ma…
docs: audit-driven URL fixes (Jammy2211 → PyAutoLabs, /release/ → /ma…
docs: audit-driven URL fixes (Jammy2211 → PyAutoLabs, /release/ → /ma…
docs: audit-driven URL fixes (workspace phase) (#7)
docs: audit-driven URL fixes (workspace phase) (#7)
docs: audit-driven URL fixes (workspace phase) (#10)
docs: audit-driven URL fixes (workspace phase) (#57)
docs: audit-driven URL fixes (workspace phase) (#70)
docs: audit-driven URL fixes (workspace phase) (#152)
Jammy2211 commentedon May 15, 2026
Workspace phase shipped + merged — all 12 PRs across libraries, tooling, and workspaces are now on main. ✅
Audit deltas:
What the remaining 104 are: mostly external paywalled / dead links we can't auto-fix (academic.oup.com journal articles, the shut-down geekfeminism.wikia.com CoC link, the bdiemer/colossus bitbucket default-branch quirk, a couple of departmental academic homepages), plus a small set of internal readthedocs page renames that need editorial decisions rather than mechanical rewrites (
pyautolens.readthedocs.io/en/latest/installation/numba.html,pyautolens.readthedocs.io/en/latest/advanced/slam.html, etc. — the docs that the link target was pointing to no longer exist; those URLs should either be removed from the docs or repointed at the new canonical page).The audit tool now lives at
admin_jammy/software/url_check/. It can be re-run any time withpython3 admin_jammy/software/url_check/url_check.py --repos <repo1> [repo2 ...](or via the conveniencerun.sh), and the report drops in alongside the script.Your originally-reported bug — the
hhttps://typo inoverview_2_new_user_guide.md— is now fixed onmainin both PyAutoLens and PyAutoGalaxy. The whole new-user-guide page's URLs have been refreshed (Jammy2211 → PyAutoLabs,/release/→/main/, and the brokennotebooks/point_source/start_here.ipynblink on the PyAutoGalaxy page is now correctlynotebooks/multi/start_here.ipynb).Closing this issue.