Repository navigation
refactor: replace os.path with pathlib across libraries and workspaces #1257
Description
Overview
Migrate all source code and workspaces from os.path.* (notably os.path.join)
to pathlib.Path. The codebase already uses pathlib extensively in newer code,
but ~99 legacy os.path.* references remain across 10 repos. This is a
mechanical refactor for consistency — no behaviour changes.
Plan
- Replace
os.path.join(a, b)withPath(a) / b - Replace
os.path.exists / isdir / isfilewithPathequivalents - Replace
os.path.dirname(os.path.realpath(__file__))withPath(__file__).parent.resolve() - Replace
os.path.basename / split / splitextwith.name / .parent / .suffix - Wrap with
str(...)only where a downstream API requires a string - Drop now-unused
import oslines - Verify per-repo with unit tests; verify workspaces with
/smoke_test
Detailed implementation plan
Affected Repositories
- PyAutoFit (primary)
- PyAutoConf
- PyAutoArray
- PyAutoGalaxy
- PyAutoLens
- autogalaxy_workspace
- autolens_workspace
- autolens_workspace_developer
- autofit_workspace_developer
- euclid_strong_lens_modeling_pipeline
Work Classification
Both (libraries first, workspaces follow)
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| PyAutoConf | main | clean |
| PyAutoFit | main | clean |
| PyAutoArray | main | clean |
| PyAutoGalaxy | main | clean |
| PyAutoLens | main | clean |
| autogalaxy_workspace | main | 88 changed (unrelated dataset regen by smoke-test-optimization) |
| autolens_workspace | main | 148 changed (unrelated dataset regen by smoke-test-optimization) |
| autolens_workspace_developer | main | TBD |
| autofit_workspace_developer | main | TBD |
| euclid_strong_lens_modeling_pipeline | main | TBD |
Suggested branch: feature/use-pathlib
Worktree root: ~/Code/PyAutoLabs-wt/use-pathlib/ (created later by /start_library)
Implementation Steps
- Refactor each library in dependency order: PyAutoConf → PyAutoFit → PyAutoArray → PyAutoGalaxy → PyAutoLens. Run unit tests after each.
- Refactor workspaces: autogalaxy_workspace, autolens_workspace, autolens_workspace_developer, autofit_workspace_developer, euclid_strong_lens_modeling_pipeline. Run
/smoke_testto validate. - Resolve workspace dirty state in autogalaxy_workspace and autolens_workspace before cutting workspace branches (coordinate with smoke-test-optimization task).
- Ship via
/ship_librarythen/ship_workspace.
Key Files
PyAutoConf/autoconf/directory_config.py—os.path.isdirchecksPyAutoConf/autoconf/setup_colab.py—os.path.joinfor workspace pathsPyAutoConf/autoconf/json_prior/generate.py—os.path.existsPyAutoConf/test_autoconf/test_fitsable.py— manyos.path.joinin test dataPyAutoFit/autofit/non_linear/paths/directory.py—os.path.exists/isfilePyAutoFit/autofit/non_linear/paths/abstract.py—os.path.existsPyAutoFit/autofit/non_linear/search/**/search.py— checkpoint file checksPyAutoFit/autofit/mapper/model_mapper.py—os.path.dirname(os.path.realpath(__file__))PyAutoFit/autofit/tools/util.py—os.path.join,os.path.split,os.path.existsPyAutoFit/autofit/database/__init__.py—os.path.existsPyAutoArray/autoarray/util/dataset_util.py,autoarray/plot/utils.py, 4 test filesPyAutoGalaxy/autogalaxy/plot/plot_utils.py,autogalaxy/util/plot_utils.py, 1 test filePyAutoLens/...— 19 references (largest by count)autolens_workspace/scripts/imaging/features/advanced/subhalo/sensitivity/slam_source_*.pyautolens_workspace/scripts/interferometer/casa_reduction.py- All
docs/conf.py(sphinx boilerplatesys.path.insert(0, os.path.abspath("."))— converted too, per "no more os.path")
Risks / edge cases
- External libraries that may need str (some FITS readers in older versions) — wrap with
str()where needed. os.path.realpathresolves symlinks;Path.resolve()is equivalent in Python 3.6+.- Test fixtures: pytest's
tmp_pathis already aPath, aligns naturally. - The 88 + 148 dirty files in autogalaxy_workspace / autolens_workspace are dataset binaries from in-progress smoke-test-optimization — handle separately at workspace-phase time.
Original Prompt
Click to expand starting prompt
Most examples now use Pathlib, but there are still some os.path.join uses from
legacy.
Can you update all source code and workspaces to not use path.join at all
and always use Pathlib?
This should aspan all source code, workspaces, etc, no more os.path!
Activity
Jammy2211 commentedon May 6, 2026
Scope correction
Initial scan only counted
os.path.*(99 refs). Re-scan including barepath.*fromfrom os import pathfinds ~776 references across the same 10 repos:os.path.*path.*Per the prompt's literal "no more os.path!", we're going for the full sweep — both forms.
Jammy2211 commentedon May 7, 2026
Smoke Test Results — 2026-05-07
* All 6 euclid scripts initially failed with
WorkspaceVersionMismatchError(workspace at 2026.4.13.6 vs library at 2026.5.1.4). This is a pre-existing issue unrelated to the pathlib refactor. WithPYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1set, all 6 pass cleanly. Recommend adding that toeuclid_strong_lens_modeling_pipeline/config/build/env_vars.yamldefaults in a follow-up.Library refactor verified — no regressions. The 5 library PRs are safe to merge.
Jammy2211 commentedon May 7, 2026
Library PRs Merged — 2026-05-07
PRs (all merged)
Status
Library refactor done. Workspace pathlib refactor still pending. Next:
/start_workspaceto convert ~149 references acrossautogalaxy_workspace,autolens_workspace,autolens_workspace_developer,autofit_workspace_developer, andeuclid_strong_lens_modeling_pipeline.Outstanding (separate fix)
The euclid pipeline currently fails the workspace-version check (workspace 2026.4.13.6 vs library 2026.5.1.4). Add
PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1toeuclid_strong_lens_modeling_pipeline/config/build/env_vars.yamldefaults — this is not a regression from this PR.Jammy2211 commentedon May 7, 2026
Shipped — 2026-05-07
PRs (all merged)
Library:
Workspace:
Summary
Replaced all
os.path.*and barepath.*(fromfrom os import path) withpathlib.Pathacross 5 libraries and 5 workspaces. ~700 references converted in total. 3,188 library unit tests pass; 36/36 workspace smoke tests pass. No public API changed — internal refactor only.Session Notes
path.*fromfrom os import path. Updated the issue with the corrected breakdown mid-task.PlotPatch.__call__now stringifies recorded paths (self.paths.append(str(path))). Library plotter code now passesPathtoplt.savefigin some paths andstrin others; uniformly storingstrin the test fixture avoids fragility in dozens of plot-output assertions. This is the only test-fixture-level change.docs/conf.pyboilerplate updated:sys.path.insert(0, str(Path(".").resolve()))instead ofos.path.abspath(".").main; intermediate Python scripts that touched them re-saved as LF, which inflated diffs by 5–10×. A targeted post-pass restored CRLF where needed before each commit.path.join(...)blocks left trailing/artefacts that the regex couldn't clean — worth a note for future skill design.Outstanding (separate fix, not regression)
The euclid pipeline currently fails its workspace-version check by default (workspace 2026.4.13.6 vs library 2026.5.1.4). With
PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1set, it passes cleanly. Recommend adding that env var toeuclid_strong_lens_modeling_pipeline/config/build/env_vars.yamldefaults in a follow-up — pre-existing issue, not caused by this PR.ensure_workspace_labels.shdoes not coverJammy2211/autofit_workspace_developerorPyAutoLabs/autolens_workspace_developer— thepending-releaselabel was missing on the former and had to be created viagh api. Worth adding both to the helper.Post-merge cleanup
~/Code/PyAutoLabs-wt/use-pathlibremoved.feature/use-pathlibbranches deleted (local + origin).main.autogalaxy_workspaceandautolens_workspacepulls skipped — their working trees have uncommitted dataset-binary changes from the in-progresssmoke-test-optimizationtask. The merged commits will land when that task ships its dirty state.refactor: replace os.path with pathlib