diff --git a/docs/README-lanes.md b/docs/README-lanes.md index 955d5fcb..c551c93e 100644 --- a/docs/README-lanes.md +++ b/docs/README-lanes.md @@ -3586,13 +3586,6 @@ place, under one rule: **nothing is deleted that is not first on origin or in a bundle under the sweeps directory, and a tree a live writer owns is never touched.** -> **`--yes` is switched off until opensoft/openRepoTools#170 lands.** Reviews -> of this act found data-loss paths under `--yes` and `--expire --yes` that are -> listed there and not yet closed. Until they are, both are refused, exit **2**, -> changing nothing, unless the environment carries `LANE_WORKTREES_ENABLE_YES=1` -> — which the test suite sets, and which is **not for a real estate until #170**. -> The dry run, `--porcelain` (#163's gate) and `--report` are unaffected. - ```sh lane-worktrees sweep # the DRY RUN: the table, nothing changed lane-worktrees sweep --yes [--live |--live none] @@ -3606,23 +3599,42 @@ also reads every registration of the lane's checkout and every checkout under its two roots (`/.claude/worktrees/*`, `.lane-worktrees//*`). A tree the inventory does not name is still **the lane's** when it stands under the lane's own root, `.lane-worktrees//` (only this lane's starts make it), -or sits in `/.claude/worktrees` on a branch whose OWN commits — those -origin's default branch lacks — carry this lane's `Lane:` trailer; its rows say -so. A lane with no #97 snapshot at all (state `NONE`) but trees of its own is +or sits in `/.claude/worktrees` with its HEAD — a commit origin's +default branch lacks — carrying this lane's `Lane:` trailer AS ITS OWN (a tree +another lane stacked on this lane's commit is not this lane's, #174); its rows +say so. A lane with no #97 snapshot at all (state `NONE`) but trees of its own is **refused, exit 2**: which trees are its is recorded nowhere, so #163's gate does not pass it. Any other tree the inventory does not name is **FOREIGN**: reported and left, unless `--include-foreign` and a `--word ""` (recorded in every register line it causes). A tree ANOTHER lane's inventory names is that lane's and is never taken from here, word or no word — and one BOTH inventories name is kept, since -records are history and a path can be reused. An inventory record that cannot be -read (no path, an unknown schema) refuses the sweep, exit **2**: its tree's owner -is unknown — and so does ANOTHER lane's record or inventory directory that cannot -be read, before anything is fetched. The lane's own checkout is never a candidate. The lane name is +records are history and a path can be reused. Other lanes' claims are read +wherever #97 keeps them (#170 G6): beside this lane's control root, under +`$LANES_LANE_STATE_ROOT` and `$PROJECTS_ROOT/.lane-state`, beside every checkout +the estate's shape walk finds, beside every lane's recorded `dir` in the +register's lane logs that lies in this sweep's estate (#174), and - where the whole estate is walked (a clone the table +could take, or `--include-scratch`) - in every `.lane-state` that walk finds, one +in a plain directory inside a repository included (#174). An inventory record that cannot be +read (no id, no path, an unknown schema; #170 G9) refuses the sweep, exit **2**: its tree's owner +is unknown — and so does ANOTHER lane's record or inventory directory, a lane +worktree root, a checkout's `git worktree list`, or the register's lane logs that +cannot be read, before anything is fetched (#170 G1, G5); another lane's claim +that cannot be read is a NOTE instead where this lane has no tree, since a claim +could only take one from it (#174). The lane's own checkout is never a candidate. The lane name is resolved first (`lanes-edit.sh canon-lane`, Amendment 15), as `lane-start` and `lane-end` resolve it; an alias table that cannot be read refuses. **Who may act** is #97's reconciliation (`lanes-edit.sh lane-reconcile`), read -before anything is written — the fetch included: +before anything is written — the fetch included. Under `--yes` the register is +FETCHED FIRST, by the sweep itself (`origin/` of the workspace +repository, an explicit refspec): `lanes-edit.sh`'s own fetch answers 0 when it +fails, and an inherited `LANES_NO_FETCH=1` skips it, so a lane rebound on another +host after this one's last fetch would read as bound HERE (#170 A1). A register +that cannot be fetched refuses, exit **2**, with nothing changed. The fetch of +each repository acted in pins its refspec as `status --fetch` does +(`+refs/heads/*:refs/remotes/origin/*`, `--refmap` the same, tag pruning off), so +a mirror-style `remote.origin.fetch` cannot let `--prune` delete a local branch +before it is classified (#170 item 6). | the lane | dry run | `--yes` | |---|---|---| @@ -3630,7 +3642,7 @@ before anything is written — the fetch included: | its binding, holder or reconciliation could not be read | the table, exit **2** | refused, exit **2** | | held by ANOTHER live session | the table, exit **2** | refused, exit **2** | | managed-owned (ruling 2026-10-04) | refused, exit **2** | refused, exit **2** | -| held by THIS session | the table | only with the coordinator's writer count: `--live ` per live writer (its ListAgents count, Amendment 17 Addendum 1 (i)), or `--live none` | +| held by THIS session, or with a tree THIS session recorded (an inventory `writer` that is the caller, #170 B5) | the table | only with the coordinator's writer count: `--live ` per live writer (its ListAgents count, Amendment 17 Addendum 1 (i)), or `--live none`. A path that names no tree of the table, or `none` beside a path, is usage, **64** (#170 B4) | | `SWAPPING` in a live session | the table | refused: a handoff is in flight | | no live holder, bound here, free or gone | the table | performed | @@ -3642,21 +3654,26 @@ Each tree is classified in this order; the first row that matches decides. |---|---|---| | not in the lane's inventory | `foreign` | nothing (`--include-foreign --word …` applies the rows below) | | a process stands in it or holds a file open there (`/proc`, else `lsof`), a tmux pane's current path is in it, it is named by `--live`, the sweep was started inside it, or the session that recorded it is live | `live` | nothing, ever — asked again at the moment of the act | -| liveness could not be read | `keep` | nothing | +| liveness could not be read — the scan failed, or a process of THIS account whose `/proc` entries cannot be read (not dumpable) is placed in it by an absolute path in its command line or by its parent's working directory (#170 item 2) | `keep` | nothing | | its directory is gone and git still registers it | `prune` | `git worktree remove ` — that one registration; a repository-wide `git worktree prune` is never run (it would unregister other lanes' and FOREIGN trees too); a directory that reappeared meanwhile is left for the next sweep to classify | +| … and its registration's git directory names a commit no branch, tag or origin holds — a detached HEAD, a reflog entry (#170 A4) | `rescue+prune` | a rescue branch pushed (where there is an origin) and a bundle, then that one registration removed | +| … and its registration's git directory keeps a submodule repository (`modules/`) with a branch, tag, stash or reflog entry no remote of it holds, or an annotated tag object none of them holds (#174) | `keep` | nothing: the prune would take that repository (asked again just before) | | its directory is gone and nothing registers it | `gone` | nothing; the inventory record is history | | another lane's inventory names it too | `keep` | nothing: which lane owns it now is a person's to say | -| a registration someone LOCKED; a submodule with uncommitted work, ignored files no commit carries (an `.env`), or a commit no remote holds; a repository nested inside it; a CLONE with a branch or a stash origin lacks, or whose git directory linked worktrees share | `keep` | nothing, and the line says which | -| something leans on it — another repository's `objects/info/alternates`, or a remote whose URL is its path | `load-bearing` | nothing; the line names every dependent and the remedy (`git repack -a -d`, then drop the alternates or re-point the remote) | +| a registration someone LOCKED; a file flagged skip-worktree or assume-unchanged that differs from the index (git status hides it, #170 A2); a submodule with uncommitted work, ignored files no commit carries (an `.env`), a branch, tag or HEAD no remote of it holds, a stash (#170 item 5), a commit only its reflog names, or a skip-worktree or assume-unchanged edit (#174) — its repository lives under the tree's git directory and goes with it; any submodule repository that git directory keeps under `modules/` with a branch, tag, HEAD, stash or reflog entry no remote of it holds, or an ANNOTATED tag whose very tag object none of them holds (asked of each remote at the act), a DEINITIALIZED submodule's included (#174); a repository nested anywhere inside it, a venv's `src/`, node_modules and a cache-named directory included (#170 A3, #174); a CLONE with a branch, a TAG (#170 item 4) or a stash origin lacks, or whose git directory linked worktrees share | `keep` | nothing, and the line says which | +| something leans on it — another repository's `objects/info/alternates`, or a remote whose URL is its path, read from every repository the estate walk finds: every directory of the estate but caches, tool environments and `site-packages`, a plain directory inside a repository included (#170 A11) | `load-bearing` | nothing; the line names every dependent and the remedy (`git repack -a -d`, then drop the alternates or re-point the remote) | +| a CLONE (removed by deleting its directory) when the search for what leans on it was partial: a directory of the estate could not be read, there is no estate root, or the clone lies outside it (#170 A11) | `keep` | nothing | | no `origin` | `keep` (`bundle+remove` with `--bundle`) | a bundle is its only rescue | -| dirty or untracked work | `wip-rescue+remove` | `git add -A` into a COPY of its index, `commit-tree` on its head, `rescue//-` pushed and seen on origin, a bundle, then removed | +| dirty or untracked work, where git-lfs is configured and an untracked file is `filter=lfs` content | `keep` | nothing: untracked files go to a bundle only (below), and a bundle carries an LFS pointer, never its bytes (#174); asked again at the act, before anything is pushed | +| dirty or untracked work | `wip-rescue+remove` | the tracked changes (`git add -u` into a COPY of its index), `commit-tree` on its head, `rescue//-` pushed and seen on origin; UNTRACKED files in a second commit on it that goes to the bundle ONLY, never to origin — an un-ignored key file is not something anyone meant to push (#170 A9; `--push-untracked` pushes them as before). Which files are untracked is read from the two snapshots taken AT THE ACT, never from the table's status, so a file that arrived since is held back too (#174); a bundle, then removed | | an unborn branch, clean | `remove` | removed | | detached at a commit origin holds | `remove` | removed | | detached with commits of its own | `rescue+remove` | `rescue//-` pushed, a bundle, then removed | -| MERGED — its PR LANDED in the register, or `gh` says MERGED, into the default branch, and that PR's head holds this tip | `remove+delete-branch` | removed; local branch deleted at the SHA judged merged (`update-ref -d `), and the remote branch too — under a lease on the judged SHA — where its `Lane:` trailer is this lane's and it holds nothing the PR did not | +| MERGED — its PR LANDED in the register, or `gh` says MERGED, into the default branch, and that PR's head holds this tip | `remove+delete-branch` | removed; local branch deleted at the SHA judged merged (`update-ref -d `), and the remote branch too — under a lease on the judged SHA — where its `Lane:` trailer is this lane's, it holds nothing the PR did not, and `gh` does not still answer a PR of it OPEN (a LANDED line with a wrong number never closes an open PR, #170 A8) | | clean, every commit on its own remote branch (an open PR is untouched) | `remove` | removed; the branch stays | | clean, its head on some origin branch | `remove` | removed; the branch stays | | unpublished commits on `main`/`master`, or on a branch origin has DIVERGED from | `rescue+remove` | the tip to `rescue//-`, then removed; the branch itself is never force-pushed | +| unpublished commits on a branch an OPEN pull request names (gh's OPEN wins over a register LANDED line naming that PR, #174), whose origin tip carries ANOTHER lane's `Lane:` trailer, or where whether a PR names it could not be read (#170 A7) | `rescue+remove` | the same: the lane's unreviewed commits never go onto somebody else's branch, and origin's branch is left as it is | | unpublished commits otherwise | `push+remove` | pushed AS IS under its OWN name — never its upstream's, which for a branch made from `origin/main` is `main` — then removed | **"Merged" is never `git branch --merged` alone**: a squash merge leaves no @@ -3674,7 +3691,30 @@ a scan that cannot be made leaves the tree. **At the moment of removal** the head is re-read, and the tree must still be clean — or, after a WIP rescue, must still be byte for byte the tree the rescue commit carries. A push that origin refuses leaves the tree exactly as it was: the WIP commit is made BESIDE the tree, never into it, so its branch, index and -files never move. +files never move. Every push is checked on origin (`git ls-remote`) before +anything goes, and where the repository has git-lfs configured (`filter.lfs.*`) +it runs its hooks, so git-lfs's pre-push uploads the objects (#170 A10). + +**Before each removal git is asked what it would take.** A worktree's removal +takes its own git directory — its HEAD and that HEAD's reflog (and its branch +and the branch's reflog, where the branch is deleted after it); a pruned +registration's the same; a CLONE takes every ref but its remote-tracking ones, +and every reflog — a reflog directory that cannot be listed makes the answer +unknown, and the clone stays (#174) — and an ANNOTATED TAG is its own object, +kept where origin holds that very tag object (`ls-remote --tags`) and bundled, +its message with it, where it does not (#174); a `--branches` delete takes the +branch's tip and its reflog (#174). Commits there that no origin ref, surviving +branch, tag or stash keeps — a commit only a reflog names, after an amend or a checkout away +from it (#170 A5) — go to a bundle first, through temporary +`refs/lane-worktrees/-/*` refs deleted once it verifies. **The +self-check** asks the same question once more right before the removal: a +commit the removal would take that is neither on origin, in a ref that stays, +in a push seen on origin this run, nor in a bundle this run verified is a +defect in the table, and the sweep REFUSES it — exit **2**, a line in +`DISPOSITION.md`, nothing after it acted on. A CLONE with a local git-lfs +store (`.git/lfs/objects`) whose commits would go to a bundle only is kept +instead: a bundle carries LFS pointers, never their bytes (#174). The first register line that +cannot be written stops the sweep the same way (exit 1, #170 B6). **Ignored files that are not caches or build output** (an `.env`, a local config) are archived to `-ignored.tar.gz` before a tree goes: a commit @@ -3682,17 +3722,21 @@ cannot carry them, and the rule is that nothing goes that is not first saved. They are listed again at the moment of removal, and a tree whose ignored files changed, appeared or went since the archive was taken is left. Build output (`target`, `build`, `dist`, `.tox`, `.next`, `*.egg-info`, …) is -generated and is not archived; more than `ignored_archive_mb` of the rest leaves -the tree in place for a person. +generated and is not archived — judged by the ignored entry's OWN name, never an +ancestor's, so `docker/build/prod.env` is archived (#170 A6) — and so are +`docker/node_modules/prod.env` and `docker/.venv/prod.env`: the archive leaves +caches out only BENEATH an entry it was given, and is read back for every entry +selected, so one missing leaves the tree (#174). More than +`ignored_archive_mb` of the rest leaves the tree in place for a person. ### The other rows | flag | what | disposition | |---|---|---| -| `--branches` | local branches no worktree holds, in the lane's checkout | merged by PR evidence: `delete` (`delete+remote` where the `Lane:` trailer is this lane's and origin's tip is inside the PR), each at the SHA judged and only after a fetch of that checkout that worked under `--yes`; an open PR's branch: `keep`; unmerged with a missing, diverged, unpushed or unreadable upstream: `list` with its tip, distance from `origin/main`, last commit date and owner — never deleted; `main`, `master` and `rescue/*`: never touched | -| `--include-scratch` | `.lane-worktrees//*-scratch`, `briefs/`, `bin/` that are no checkout | `archive+remove`: tar (caches left out) and sha256, then removed — only if no writer arrived and nothing outside the caches changed while the tar ran | -| `--include-caches` | `__pycache__`, `.pytest_cache`, `.mypy_cache`, `.ruff_cache`, `node_modules`, any directory with `pyvenv.cfg` (and `venv/`, `.venv/` with an activate script), under `.lane-worktrees//**` and the lane's inventory trees | `remove`, without archiving — never one git tracks or whose tracking cannot be read, one inside scratch being archived, or one in a FOREIGN tree, a live one, one whose liveness is unknown or one left by its own act; the owning tree's liveness is asked again just before | -| `--include-sandboxes` | `tmp.*` and `pytest-of-$USER/pytest-*` in `/tmp` and `$TMPDIR` (or `LANE_WORKTREES_SANDBOX_ROOTS`), owned by this account | `remove` where the owning process is gone: a pytest `.lock` naming a dead pid (read again at the act), and no live process standing in it, holding it open or naming it in its environment; a `tmp.*` younger than `sandbox_min_age_minutes` is left | +| `--branches` | local branches no worktree holds, in the lane's checkout | merged by PR evidence: `delete` (`delete+remote` where the `Lane:` trailer is this lane's, origin's tip is inside the PR and gh does not answer it OPEN), each at the SHA judged and only after a fetch of that checkout that worked under `--yes`, and only after what the delete would take — its tip and every commit its reflog names — is on origin, in a ref that stays or in a bundle (a `bundle:` ledger row), asked once more right before the delete: the self-check, exit **2** on a refusal (#174); an open PR's branch: `keep` — and still COUNTED where it is the lane's and `origin/` lacks its tip (#170 G2); a branch `origin/` holds, whatever its configured upstream, is published and not listed (#170 G7); unmerged with a missing, diverged, unpushed or unreadable upstream: `list` with its tip, distance from `origin/main`, last commit date and owner — never deleted; `main`, `master` and `rescue/*`: never touched | +| `--include-scratch` | `.lane-worktrees//*-scratch`, `briefs/`, `bin/` that are no checkout | `archive+remove`: tar (caches left out) and sha256, then removed — only if no writer arrived and nothing outside the caches changed while the tar ran; `keep` while a tree of the table or any `.git` lies under it (#170 item 7), or the search for what leans on it was partial (#170 A11) | +| `--include-caches` | `__pycache__`, `.pytest_cache`, `.mypy_cache`, `.ruff_cache`, `node_modules`, any directory with `pyvenv.cfg` (and `venv/`, `.venv/` with an activate script), under `.lane-worktrees//**` and the lane's inventory trees | `remove`, without archiving — never one git tracks or whose tracking cannot be read, one that is or holds a repository or a directory that cannot be read (`keep`: a venv's `src/` clone is work, #174; asked again just before), one inside scratch being archived, or one in a FOREIGN tree, a live one, one whose liveness is unknown or one left by its own act; the owning tree's liveness is asked again just before | +| `--include-sandboxes` | `tmp.*` and `pytest-of-$USER/pytest-*` in `/tmp` and `$TMPDIR` (or `LANE_WORKTREES_SANDBOX_ROOTS`), and `tests/run.sh`'s run roots, owned by this account | `remove` where the owning process is gone: a pytest `.lock` naming a dead pid (read again at the act), and no live process standing in it, holding it open or naming it in its environment. A `tmp.*` is NEVER removed (#170 item 1, option (b); #174): `mktemp -d` records no owner, so neither a suite's mark in it nor `aging_days` untouched proves a suite made it — one older than `sandbox_min_age_minutes` (by the newest file in it) is `list`ed and left, with its mark and any repository in it (a bare one included), since it may be a person's checkout or scratch | | `--links` | every symlink under the estate and every worktree `.git` gitdir pointer | `broken`, listed with target and age; nothing changes | | `--bundle` | every tree acted on | a `git bundle` beside the archive (always, for a rescue) | @@ -3705,7 +3749,9 @@ started in the same second gets `-2`) and never by a dry run: * `DISPOSITION.md` — every tree and item acted on: what it was, why, what was done, where its rescue is, and the register line that records it; * `rescues.tsv` — `origin`, rescue branch and sha for each rescue, which - `--expire` reads; + `--expire` reads; a bundle that is the ONLY copy of what it holds (untracked + files, commits only a reflog named, a tree with no origin) is a row too, with + origin `-` and branch `bundle:` (#170 item 8); * the bundles, `-ignored.tar.gz` and `scratch-.tar.gz`; * `MANIFEST.sha256` — `sha256sum` format, every file above. @@ -3719,8 +3765,8 @@ outlives the archive. **Retention: 90 days** (`sweep.conf`). `sweep --expire` lists archives older than that, and with `--yes` removes those whose every rescue branch is still on origin AT THE RESCUED SHA (`git ls-remote`); an archive whose rescue branch is -gone from origin, or was remade at another commit, is the only copy and is NEVER -expired. Nor is an archive that is not WHOLE — no `MANIFEST.sha256`, a file it +gone from origin, or was remade at another commit, or that holds a `bundle:` row, +is the only copy and is NEVER expired. Nor is an archive that is not WHOLE — no `MANIFEST.sha256`, a file it does not list or whose digest differs (a sweep interrupted after a bundle), or a `rescues.tsv` that is missing or has a row that is not origin, branch and SHA, or a manifest entry whose file is gone. What expired is appended to @@ -3758,12 +3804,15 @@ command exits **0** (nothing to retire), **3** (something to retire — a tree o lane's on disk, live or not, a stale registration, an unpublished or merged branch of the lane's, included scratch), or **2** (refused: bound elsewhere, held by another session, managed, a lane with trees and no #97 snapshot, a read -failed, `--yes` switched off — or an error no read caught, which is never a -traceback's exit 1 on a dry run). FOREIGN trees, links, +failed — a lane root, a registration list, another lane's claim on a lane with +a tree, the register's lane logs, or under `--yes` the register's fetch — or an +error no read caught, which is never a traceback's exit 1 on a dry run). FOREIGN trees, links, caches and sandboxes are never the lane's to retire. `--yes` exits 0 when every -act completed and 1 when one failed part-way (the rest go on; each failure is in +act completed, 1 when one failed part-way (the rest go on — unless a register +line could not be written, which stops the rest; each failure is in the table and in `DISPOSITION.md`, which is written even when an error no read -caught stops the run); usage is 64. +caught stops the run), and **2** when the self-check refused a removal; usage +is 64, a `--live` path that names no tree included. ### Two protocol lines this act assumes (proposed for the amendment that ratifies it) @@ -3855,12 +3904,14 @@ The sweep retires what is already there; these keep more from arriving. repeated). Every commit `lanes-edit.sh` makes asks git what its pathspec would stage (`git add --dry-run`) or holds staged already, and REFUSES, exit **2**, nothing staged, when any of it is bytecode, a cache, a dependency tree or a virtual environment — and - offers the `.gitignore` lines a workspace without them lacks. Attachments are + offers the `.gitignore` lines a workspace without them lacks. A pathspec git + cannot read (a malformed magic pathspec, an unreadable index) is refused the + same way, never read as clean (#170 G13). Attachments are committed by hand, so the same question is a subcommand: ```sh lanes-edit.sh pathspec-check handoffs//attachments/ - # 0 clean · 2 the offending paths on stdout, the offer on stderr · 64 usage + # 0 clean · 2 the offending paths on stdout, the offer on stderr, or the read failed · 64 usage ``` ## Hand edits diff --git a/lane-worktrees b/lane-worktrees index 8fd16fb9..66afa339 100755 --- a/lane-worktrees +++ b/lane-worktrees @@ -5,7 +5,7 @@ abandoned worktrees, branches, scratch and killed-suite sandboxes (opensoft/openRepoTools#162). lane-worktrees sweep [--dry-run | --yes] [--porcelain] - [--bundle] [--branches] [--links] + [--bundle] [--branches] [--links] [--push-untracked] [--include-scratch] [--include-caches] [--include-sandboxes] [--include-foreign --word ""] [--live ... | --live none] [--fetch] [--estate ] @@ -23,8 +23,30 @@ live writer owns is never touched. THE DRY RUN IS THE DEFAULT and changes nothing - no fetch, no ref, no file, no register line (`--fetch` asks origin first; `lanes-edit.sh`'s own reads -fetch the register as they always do). `--yes` performs the table, fetching -origin first in every repository it acts in. +fetch the register as they always do). `--yes` performs the table: it fetches +the register itself first and refuses if that fails (an inherited +LANES_NO_FETCH does not skip it), then fetches origin in every repository it +acts in, with a pinned refspec that never prunes a local branch or tag. + +BEFORE EACH REMOVAL git is asked which commits it would take - a worktree's +HEAD and reflog, a clone's every ref (an annotated tag's object included) +and reflog, a pruned registration's, a `--branches` delete's tip and reflog - +and any that no origin ref, surviving branch, tag or stash keeps go to a +bundle first (ONLY-COPY bundles are rows of rescues.tsv, and `--expire` +never removes their archive). The same question is the SELF-CHECK right +before the removal: anything neither on origin nor in a verified bundle is +refused, exit 2, and nothing after it is acted on; so is everything after a +register line that could not be written. Untracked files are never pushed +(the bundle only) unless `--push-untracked`, and untracked git-lfs content, +which a bundle would carry as a pointer only, keeps its tree; a branch an open +pull request or another lane owns is never pushed onto (a rescue branch +instead, and gh's OPEN wins over a LANDED line); where git-lfs is configured +the push runs its hooks, and a clone whose local git-lfs store a bundle alone +would have to stand for is kept. A cache that holds a repository (a bare one +included) is never removed, nor is any `tmp.*` sandbox (it is listed: nothing +proves a suite made it), nor a tree or registration whose git directory keeps +a submodule repository (a deinitialized one's included) with work no remote +of it holds. WHICH TREES ARE THE LANE'S is #97's inventory (`lanes-edit.sh lane-trees`) and the disk - never the derived index (Amendment 14 clause (b)). A tree the @@ -35,8 +57,10 @@ that lane's and is never acted on from here. WHO MAY ACT is #97's reconciliation (`lanes-edit.sh lane-reconcile`): a lane bound on another host or container is UNKNOWN, never dead (Amendment 18(b)), so every act is refused for it; a lane held by another live session is -refused; a lane whose holder is THIS session acts only on the writer count -the coordinator states with `--live` (Amendment 17 Addendum 1 (i)). +refused; a lane whose holder is THIS session, or one with a tree THIS session +recorded, acts only on the writer count the coordinator states with `--live` +(Amendment 17 Addendum 1 (i)) - each path naming a tree of the table, or +`--live none` alone (anything else is usage, 64). "MERGED" is the register's LANDED line for the branch's pull request, or `gh`'s MERGED state for it - never `git branch --merged` alone, because a @@ -47,8 +71,9 @@ sweeps/// with MANIFEST.sha256, DISPOSITION.md and rescues.tsv. `sweep --expire` lists (and with `--yes` removes) archives older than the retention (90 days; `sweep.conf`) whose every rescue branch is still on origin at the rescued SHA; an archive whose rescue branch is gone or moved, -or that is not whole (its MANIFEST.sha256 missing or not matching, its -rescues.tsv unreadable), is never expired. +that holds an only-copy bundle (a `bundle:` row), or that is not whole (its +MANIFEST.sha256 missing or not matching, its rescues.tsv unreadable), is +never expired. THE ESTATE REPORT (`--all --dry-run --report`) is the net under every actor that never runs `lane-end`: FOREIGN clones, trees of ENDED lanes, unmerged @@ -60,11 +85,6 @@ file or an issue. It changes nothing. `lane-start` runs it once a day per workstation, detached (`LANE_WORKTREES_REPORT=off` is the switch). It exits 0, or 1 when `--post` could not deliver it (the report is then printed). -`--yes` (and `--expire --yes`) is SWITCHED OFF until opensoft/openRepoTools -#170 lands: it is refused, exit 2, unless the environment carries -LANE_WORKTREES_ENABLE_YES=1 - which the test suite sets, and which is not for a -real estate until #170. The dry run, `--porcelain` and `--report` are whole. - PORCELAIN: one row per line, fields separated by one TAB. In every field a backslash is written `\\`, a TAB `\t`, a newline `\n` and a carriage return `\r`, so a path holding any of them is still one field of one row. @@ -74,8 +94,11 @@ Exit codes: 0 dry run: nothing to retire · --yes: every act completed 3 dry run: something to retire (the `lane-end` gate's answer, #163) 2 refused: the lane is bound elsewhere, held by another live session, - managed-owned, has trees but no #97 snapshot, or a read failed - or - `--yes` is switched off, or an error no read caught - nothing was changed + managed-owned, has trees but no #97 snapshot, or a read failed (a lane + root, a registration list, another lane's claim on a lane with a tree, + the register's fetch under --yes) - or an error no read caught - + nothing was changed; or, under --yes, the self-check refused a removal + (what was done before it is recorded, nothing after it was acted on) 1 --yes: an act failed part-way; what was done is recorded and the rest is left exactly as it was 64 usage @@ -203,8 +226,20 @@ def run(argv, cwd=None, env=None, timeout=GIT_TIMEOUT, input_text=None): return subprocess.CompletedProcess(argv, 127, "", str(exc)) -def git_env() -> dict: +#: Repositories read as a git directory ALONE (#174, Copilot round 2): a +#: submodule repository under `/modules/` names its working tree in +#: `core.worktree`, and where that directory is gone - a gone worktree's +#: registration, a submodule removed after its deinit - every git command +#: run in it dies on "cannot chdir". Only reads that need no working tree are +#: run there (refs, reflogs, rev-list, ls-remote), so each is handed the +#: repository's own `objects/` as one. See `module_reads`. +_GITDIR_READS: set = set() + + +def git_env(path: str = "") -> dict: env = dict(os.environ) + if path and path in _GITDIR_READS: + env["GIT_WORK_TREE"] = os.path.join(path, "objects") env["GIT_TERMINAL_PROMPT"] = "0" env.setdefault("GIT_SSH_COMMAND", "ssh -o BatchMode=yes") env["LC_ALL"] = "C" @@ -215,7 +250,7 @@ def git_env() -> dict: def git(path: str, *args: str, timeout: int = GIT_TIMEOUT): - return run(["git", "-C", path, *args], env=git_env(), timeout=timeout) + return run(["git", "-C", path, *args], env=git_env(path), timeout=timeout) def git_out(path: str, *args: str): @@ -470,8 +505,15 @@ class Helper: facts.inventory_rc = tproc.returncode if tproc.returncode == 0: for line in tproc.stdout.splitlines(): + if not line.strip(): + continue fields = line.split(US) if len(fields) < 2 or not fields[0]: + # A ROW WITH NO ID IS AN UNREADABLE RECORD, never no record + # (#170 G9): skipped, a tree it names outside the scanned + # roots went unseen and the dry run could exit 0. + facts.inventory.append({"id": "", "path": "", "schema": "", + "raw": line}) continue keys = ("id", "path", "branch", "head", "upstream", "dirty", "unpushed", "writer", "observed", "checkout", "generation", "operation", @@ -543,6 +585,28 @@ class Helper: proc = self.call("workspace-root") return proc.stdout.strip() if proc.returncode == 0 else "" + def fetch_register(self) -> str: + """`--yes` ACTS ON A REGISTER READ NOW, or not at all (#170 A1): '' once + the workspace repository's `origin/` is fetched, else why not. + + `lanes-edit.sh`'s own `log_sync` answers 0 on every way it can fail to + fetch, and an inherited `LANES_NO_FETCH=1` skips the fetch outright, so a + lane rebound on another host after this one's last fetch read as bound + HERE and its trees were removed (Amendment 18(b)). The fetch is made + directly, with an explicit refspec, and every later helper call reads + the ref it left (`LANES_NO_FETCH=1`, set by `call` once `_fetched`).""" + root = self.workspace_root() + if not root: + return ("the workspace repository could not be found (`lanes-edit.sh " + "workspace-root` answered nothing)") + branch = os.environ.get("LANES_BRANCH", "") or "main" + proc = git(root, "fetch", "--quiet", "--no-tags", "origin", + f"+refs/heads/{branch}:refs/remotes/origin/{branch}", timeout=GIT_NET_TIMEOUT) + if proc.returncode != 0: + return (f"`git -C {root} fetch origin {branch}` failed: " + _first_line(proc.stderr)) + self._fetched = True + return "" + def transcript_live(self, uuid: str, fresh: bool = False) -> str: """live | none | unknown - one live process carrying that transcript, asked once per session however many trees it recorded (`fresh` asks @@ -558,6 +622,55 @@ class Helper: env_extra={"LANES_LANE": lane}, fetch=False) +LANE_LINE_RE = re.compile(r"^(?:STARTED|PAUSED|RESUMED|ENDED|RETIRED) — lane [^,]+, " + r"session [^,]+, [^,]+, (.*)$") + + +def payload_field(payload: str, name: str) -> str: + """One ` ` sub-field of a register payload, as + `lanes-edit.sh`'s `payload_subfield` reads it: the `; `-separated part + that starts with the name, its value up to the first space - or, quoted, + up to the closing quote.""" + want = name + " " + for part in payload.split("; "): + if not part.startswith(want): + continue + value = part[len(want):].strip() + if value.startswith('"'): + end = value.find('"', 1) + if end > 0: + return value[1:end] + return value.split(" ", 1)[0] + return "" + + +def register_lane_dirs(helper) -> tuple: + """({lane: its recorded dir}, why-not) - every lane log's last lane-kind + line that carries a `dir`, read from the workspace repository's + `origin/` in one `git grep`. Where #97 keeps that lane's control + root (`/.lane-state/`) follows from it.""" + root = helper.workspace_root() if helper.path else "" + if not root: + return {}, "the workspace repository could not be found" + branch = os.environ.get("LANES_BRANCH", "") or "main" + proc = git(root, "grep", "-E", "-e", r"^(STARTED|PAUSED|RESUMED|ENDED|RETIRED) — lane ", + f"origin/{branch}", "--", "lanes/log") + if proc.returncode not in (0, 1): + return {}, (f"the register's lane logs on origin/{branch} could not be read: " + + _first_line(proc.stderr)) + out: dict = {} + for line in proc.stdout.splitlines(): + _ref, _, rest = line.partition(":") + fname, _, text = rest.partition(":") + if not fname.endswith(".md"): + continue + m = LANE_LINE_RE.match(text) + found = payload_field(m.group(1), "dir") if m else "" + if found.startswith("/"): + out[os.path.basename(fname)[:-3]] = found + return out, "" + + def _first_line(text: str) -> str: for line in (text or "").splitlines(): if line.strip(): @@ -573,10 +686,22 @@ class ProcScan: (macOS), and every tmux pane's current path beside either. A scan that could not be made is `available = False`, and a tree whose - liveness rests on it is never acted on - not "nobody is there".""" + liveness rests on it is never acted on - not "nobody is there". + + A PROCESS OF THIS ACCOUNT WHOSE `/proc` ENTRIES CANNOT BE READ (#170 item + 2) - not dumpable, as `ssh-agent` makes itself - is in `unknown`, never + read as absent. Which tree it stands in cannot be read either, so it is + placed by what can be: an absolute path in its command line, and its + parent's working directory (a process starts where its parent stands). + A tree either places it in is of unknown liveness and is never acted + on. One that places it nowhere (an `ssh-agent` daemon, re-parented to + init) holds no tree; one whose status cannot be read at all could be in + any. A process of ANOTHER account is never a writer here and is passed + over.""" def __init__(self): self.entries: list = [] # (pid, comm, kind, path) + self.unknown: list = [] # (pid, comm, [paths it is placed by], anywhere) self.available = False self.why = "" self._env_loaded = False @@ -584,23 +709,44 @@ class ProcScan: def _scan(self) -> None: me = os.getpid() + uid = os.getuid() if os.path.isdir("/proc/self/fd"): self.available = True + pending = [] # (pid, comm, ppid, hints, anywhere) for name in os.listdir("/proc"): if not name.isdigit() or int(name) == me: continue base = "/proc/" + name - try: - with open(base + "/comm", encoding="utf-8", errors="replace") as fh: - comm = fh.read().strip() - except OSError: + status = _proc_status(base) + if status == {}: + continue # it exited + if status is None: + # NOT EVEN ITS STATUS: whose it is comes from the entry's own + # owner, and where it stands cannot be placed at all. + try: + if os.stat(base).st_uid == uid: + pending.append((int(name), "?", 0, [], True)) + except OSError: + pass continue + if status.get("State", "").startswith("Z"): + continue # a zombie stands nowhere + ids = (status.get("Uid", "").split() + ["-1", "-1"])[:2] + ours = str(uid) in ids + comm = status.get("Name", "?") + hidden = False try: self.entries.append((int(name), comm, "cwd", _proc_path(os.readlink(base + "/cwd")))) + except (FileNotFoundError, ProcessLookupError): + continue + except PermissionError: + hidden = True except OSError: pass try: fds = os.listdir(base + "/fd") + except PermissionError: + hidden, fds = True, [] except OSError: fds = [] for fd in fds: @@ -610,6 +756,14 @@ class ProcScan: continue if target.startswith("/"): self.entries.append((int(name), comm, "fd", _proc_path(target))) + if hidden and ours: + ppid = int(status.get("PPid", "0") or 0) if status.get("PPid", "").isdigit() else 0 + pending.append((int(name), comm, ppid, _cmdline_paths(base), False)) + cwd_of = {pid: where for pid, _c, kind, where in self.entries if kind == "cwd"} + for pid, comm, ppid, hints, anywhere in pending: + if ppid in cwd_of: + hints = hints + [cwd_of[ppid]] + self.unknown.append((pid, comm, hints, anywhere)) elif shutil.which("lsof"): proc = run(["lsof", "-nP", "-w", "-F", "pcfn", "-u", str(os.getuid())], timeout=120) if proc.stdout: @@ -679,11 +833,60 @@ class ProcScan: out.append((pid, comm, kind, where)) return out + def unknown_for(self, path: str) -> str: + """Why a process of this account that `/proc` would not show could be + in `path`, or '' (see the class: item 2).""" + root = real(path) + for pid, comm, hints, anywhere in self.unknown: + if anywhere: + return (f"process {pid} of this account could not be read at all (not even its " + "status), so it could be in any tree") + for h in hints: + if h == root or h.startswith(root.rstrip("/") + "/"): + return (f"process {pid} ({comm}) of this account could not be read (it is " + f"not dumpable), and {h} - its command line or its parent's " + "directory - places it here") + return "" + def _proc_path(target: str) -> str: return target[:-10] if target.endswith(" (deleted)") else target +def _proc_status(base: str): + """`/proc//status` as {field: value}; {} where the process is gone, + None where it is there and cannot be read.""" + try: + with open(base + "/status", encoding="utf-8", errors="replace") as fh: + text = fh.read() + except (FileNotFoundError, ProcessLookupError): + return {} + except OSError: + return None + out = {} + for line in text.splitlines(): + key, sep, value = line.partition(":") + if sep: + out[key.strip()] = value.strip() + return out + + +def _cmdline_paths(base: str) -> list: + """The absolute paths in a process's command line (`--x=/p` included).""" + try: + with open(base + "/cmdline", "rb") as fh: + raw = fh.read() + except OSError: + return [] + out = [] + for arg in raw.split(b"\0"): + text = arg.decode("utf-8", "surrogateescape") + for piece in [text] + text.split("=", 1)[1:]: + if piece.startswith("/"): + out.append(real(piece)) + return out + + def pid_alive(pid: int) -> bool: if pid <= 0: return False @@ -801,6 +1004,10 @@ class MergeEvidence: if row.get("state") == "OPEN" and (slug.lower(), int(row.get("number", 0))) not in landed: return {"state": "open", "pr": int(row["number"]), "source": "gh", "why": f"PR #{row['number']} is open"} + # ... AND WHILE GH STILL ANSWERS OPEN, THE REMOTE BRANCH STAYS (#170 A8): + # a LANDED line that names the wrong number (#21 for #20) must not + # delete the branch of a PR that is still open, which would close it. + gh_open = [int(r.get("number", 0)) for r in mine if r.get("state") == "OPEN"] defaults = self.default_branches(repo) elsewhere = None for row in sorted(mine, key=lambda r: -int(r.get("number", 0))): @@ -813,7 +1020,7 @@ class MergeEvidence: base = row.get("baseRefName") or "" if base not in defaults: elsewhere = elsewhere or { - "state": "elsewhere", "pr": number, "source": "gh", + "state": "elsewhere", "pr": number, "source": "gh", "gh_open": gh_open, "why": f"PR #{number} was merged into {base or 'a base gh did not name'}, " f"not {'/'.join(sorted(defaults))}: not a landing"} continue @@ -823,10 +1030,15 @@ class MergeEvidence: source = "register LANDED" if by_register else "gh MERGED" if contained: return {"state": "merged", "pr": number, "source": source, "head": head, - "why": f"PR #{number} is merged ({source}) and holds this tip"} + "gh_open": gh_open, + "why": f"PR #{number} is merged ({source}) and holds this tip" + + (f"; gh still answers PR #{gh_open[0]} OPEN, so its remote " + "branch is kept" if gh_open else "")} return {"state": "moved", "pr": number, "source": source, "head": head, + "gh_open": gh_open, "why": f"PR #{number} is merged ({source}) but this branch has moved " - f"past its head {short(head)}"} + f"past its head {short(head)}" + + (f"; gh still answers PR #{gh_open[0]} OPEN" if gh_open else "")} if elsewhere: return elsewhere return {"state": "closed", "pr": int(mine[0].get("number", 0)), "source": "gh", @@ -872,6 +1084,64 @@ def estate_repos(root: str, max_depth: int = 7) -> list: return found +#: What the estate walk never enters: caches, tool environments and installed +#: packages. None of them holds a clone anything leans on, and node_modules +#: alone is most of an estate's directories. +WALK_SKIP = frozenset(CACHE_NAMES) | {".tox", ".nox", "site-packages"} + + +class EstateWalk: + """Every repository and every `.lane-state` directory under one root, + and every directory the walk could not read.""" + + def __init__(self, root: str): + self.root = root + self.repos: list = [] + self.lane_states: list = [] + self.unread: list = [] + + +def walk_estate(root: str) -> EstateWalk: + """A walk of EVERY directory under `root` but the caches (#170 A11): + `estate_repos` descends into a repository only through worktree + containers, so a clone in a plain directory inside a repository (all of + `xFactory/xFactories/*` on Eagle) was never seen, and nor was anything + that leaned on a lane's clone from there. A directory that cannot be read + is named in `unread`, so a caller can tell a complete answer from a + partial one. Symlinks are never followed; a `.git` is never entered.""" + walk = EstateWalk(root) + if not root or not os.path.isdir(root): + walk.unread.append(f"the estate root {root or '(none)'} is not a directory") + return walk + stack = [root] + while stack: + d = stack.pop() + try: + with os.scandir(d) as it: + entries = list(it) + except FileNotFoundError: + continue + except OSError as exc: + walk.unread.append(f"{d} ({exc.strerror or exc})") + continue + for e in entries: + if e.name == ".git": + walk.repos.append(d) + continue + try: + if e.is_symlink() or not e.is_dir(follow_symlinks=False): + continue + except OSError: + continue + if e.name == ".lane-state": + walk.lane_states.append(e.path) + elif e.name not in WALK_SKIP: + stack.append(e.path) + walk.repos.sort() + walk.lane_states.sort() + return walk + + def git_common_dir(repo: str) -> str: dotgit = os.path.join(repo, ".git") if os.path.isdir(dotgit): @@ -896,13 +1166,18 @@ def git_common_dir(repo: str) -> str: return gitdir -def local_remote_urls(common: str) -> list: - """(name, url) for every remote in that config whose URL is a local path.""" +def local_remote_urls(common: str, unread: list | None = None) -> list: + """(name, url) for every remote in that config whose URL is a local path. + A config that exists and cannot be read is named in `unread`.""" out = [] try: with open(os.path.join(common, "config"), encoding="utf-8", errors="replace") as fh: text = fh.read() - except OSError: + except FileNotFoundError: + return out + except OSError as exc: + if unread is not None: + unread.append(f"{os.path.join(common, 'config')} ({exc.strerror or exc})") return out section = "" for raw in text.splitlines(): @@ -921,14 +1196,25 @@ def local_remote_urls(common: str) -> list: return out -def dependents_map(repos: list) -> dict: +def dependents_map(repos: list, unread: list | None = None) -> dict: """{physical path a repository LEANS ON: [(dependent, how)]} - read from `objects/info/alternates` and from remotes whose URL is a local path. A - clone something leans on is LOAD-BEARING and is never removed by a sweep.""" + clone something leans on is LOAD-BEARING and is never removed by a sweep. + A repository whose git directory, alternates or config exists and cannot + be read is named in `unread`: its leaning is unknown, never none.""" deps: dict = {} for repo in repos: common = git_common_dir(repo) if not common: + # A `.git` FILE THAT IS NO POINTER IS NO REPOSITORY: uv keeps an empty + # one in its cache to stop git looking upward. Only one that cannot + # be read leaves the leaning unknown. + try: + with open(os.path.join(repo, ".git"), "rb") as fh: + fh.read(1) + except OSError as exc: + if unread is not None: + unread.append(f"{repo}/.git ({exc.strerror or exc})") continue try: with open(os.path.join(common, "objects", "info", "alternates"), @@ -941,9 +1227,12 @@ def dependents_map(repos: list) -> dict: alt = os.path.normpath(os.path.join(common, "objects", alt)) target = real(alt) deps.setdefault(target, []).append((repo, "alternates " + alt)) - except OSError: + except FileNotFoundError: pass - for name, url in local_remote_urls(common): + except OSError as exc: + if unread is not None: + unread.append(f"{common}/objects/info/alternates ({exc.strerror or exc})") + for name, url in local_remote_urls(common, unread): path = url[len("file://"):] if url.startswith("file://") else url path = os.path.expanduser(path) if not os.path.isabs(path): @@ -1004,6 +1293,7 @@ class Tree: self.ahead = 0 self.behind = 0 self.lane_trailer = "" + self.remote_trailer = "" # the Lane: trailer of origin/'s tip self.merge: dict = {} self.live = None # (True|None, why) | None = not live self.lean: list = [] @@ -1018,11 +1308,20 @@ class Tree: self.ignored_archive = "" self.register = "" self.contested = False # another lane's inventory names it too + self.reg: dict = {} # its registration, where git has one + self.held_back = "" # the bundle-only commit of its untracked files + self.lost_bundle = "" # the bundle of commits only its reflog named @property def slice(self) -> str: return safe_ref_part(os.path.basename(self.path.rstrip("/"))) + @property + def gitpath(self) -> str: + """Where git is asked about it: the tree, or - its directory gone - + the checkout that registers it.""" + return self.path if self.exists else (self.checkout or self.path) + def label(self) -> str: if not self.exists: return "(gone)" @@ -1035,7 +1334,10 @@ class Tree: return f"{self.branch} {short(self.head)}" -def worktree_registrations(checkout: str) -> list: +def worktree_registrations(checkout: str): + """Every registration of `checkout`'s repository, its main worktree first + - or None where git could not list them, which is never "none" (#170 G5): + an empty answer let discovery report nothing to retire and exit 0.""" # `-z` WHERE GIT HAS IT (2.36+): a path with a newline in it is one field, # not a registration of half its name and a line of noise (B3). proc = git(checkout, "worktree", "list", "--porcelain", "-z") @@ -1044,7 +1346,7 @@ def worktree_registrations(checkout: str) -> list: else: proc = git(checkout, "worktree", "list", "--porcelain") if proc.returncode != 0: - return [] + return None fields = proc.stdout.splitlines() out, cur = [], {} for line in fields + [""]: @@ -1120,6 +1422,10 @@ def read_tree(tree: Tree) -> None: tree.remote_branch = target tip = git_out(tree.path, "rev-parse", "-q", "--verify", f"refs/remotes/origin/{target}") tree.remote_tip = tip or "" + if tree.remote_tip: + rmsg = git_out(tree.path, "log", "-1", "--format=%B", tree.remote_tip) or "" + rtrailers = re.findall(r"^Lane:\s*(\S+)\s*$", rmsg, re.MULTILINE) + tree.remote_trailer = rtrailers[-1] if rtrailers else "" if tree.head: if tree.remote_tip: counts = git_out(tree.path, "rev-list", "--left-right", "--count", @@ -1141,10 +1447,11 @@ def read_tree(tree: Tree) -> None: tree.lane_trailer = trailers[-1] if trailers else "" -def snapshot_tree(path: str) -> str: - """The tree object `git add -A` would commit in `path` NOW, written - through a COPY of its index so the tree's own index is never touched. - Empty where it could not be made.""" +def snapshot_tree(path: str, tracked_only: bool = False) -> str: + """The tree object `git add -A` would commit in `path` NOW (`git add -u` + with `tracked_only`: no untracked file), written through a COPY of its + index so the tree's own index is never touched. Empty where it could + not be made.""" index = git_out(path, "rev-parse", "--git-path", "index") if index is None: return "" @@ -1159,7 +1466,8 @@ def snapshot_tree(path: str) -> str: os.unlink(tmp) env = git_env() env["GIT_INDEX_FILE"] = tmp - if run(["git", "-C", path, "add", "-A"], env=env).returncode != 0: + if run(["git", "-C", path, "add", "-u" if tracked_only else "-A"], + env=env).returncode != 0: return "" proc = run(["git", "-C", path, "write-tree"], env=env) return proc.stdout.strip() if proc.returncode == 0 else "" @@ -1172,8 +1480,9 @@ def snapshot_tree(path: str) -> str: def tree_guards(tree: Tree) -> str: """What makes a tree unsafe to remove whatever its git state says: work a - commit in it cannot carry (a submodule's own dirt or commits, a clone - nested inside it) and a registration someone locked.""" + commit in it cannot carry (a submodule's own dirt, commits, branches, tags + or stash; a clone nested inside it; an edit git status hides) and a + registration someone locked.""" if tree.locked: return ("its worktree registration is LOCKED - someone pinned it on purpose; " f"`git -C {tree.checkout or tree.path} worktree unlock {tree.path}` is theirs to run") @@ -1190,66 +1499,352 @@ def tree_guards(tree: Tree) -> str: return (f"it is a CLONE whose git directory {len(linked)} linked worktree(s) share " f"(e.g. {linked[0]}); its removal would take their repository with it - " "remove or move them first") - # A CLONE TAKES ITS BRANCHES AND ITS STASH WITH IT, which a worktree - # never does: every one of them must be on origin first. - heads = git(tree.path, "for-each-ref", "--format=%(refname:short) %(objectname)", - "refs/heads/") - if heads.returncode != 0: - return "its branches could not be read" - for line in heads.stdout.splitlines(): - name, _, sha = line.partition(" ") - on = git(tree.path, "for-each-ref", "--count=1", "--contains", sha, - "--format=%(refname)", "refs/remotes/origin/") - if on.returncode != 0 or not on.stdout.strip(): - return (f"it is a CLONE and its branch {name} holds commits origin lacks; " - "a clone's removal takes every branch with it") + # A CLONE TAKES EVERY BRANCH, TAG AND STASH WITH IT, which a worktree + # never does: each one must be on origin first - a TAG included + # (#170 item 4), and a ref that cannot be checked keeps it. + why = unpublished_ref(tree.path, "--remotes=origin", "origin") + if why: + return (f"it is a CLONE and {why}; a clone's removal takes every branch and tag " + "with it") stash = git(tree.path, "stash", "list") if stash.returncode != 0 or stash.stdout.strip(): return "it is a CLONE with a stash, which its removal would take with it" + # AN EDIT GIT STATUS DOES NOT SHOW (#170 A2): a file flagged + # skip-worktree or assume-unchanged that differs from the index reads as + # clean, and `git add -A` skips it, so no rescue would carry it. + hidden = hidden_edits(tree.path) + if hidden is None: + return ("its skip-worktree and assume-unchanged flags could not be read, so whether it " + "holds an edit git status hides is unknown") + if hidden: + return (f"{len(hidden)} file(s) flagged skip-worktree or assume-unchanged differ from " + f"the index (e.g. {hidden[0]}): git status hides them and no rescue would carry " + "them - `git update-index --no-skip-worktree --no-assume-unchanged `, " + "then sweep again") sub = git(tree.path, "submodule", "status", "--recursive") - if sub.returncode == 0: - for line in sub.stdout.splitlines(): - if not line.strip() or line.startswith("-"): + if sub.returncode != 0: + return ("its submodules could not be listed (git submodule status failed), so whether " + "they hold work its removal would take is unknown") + subs = set() + for line in sub.stdout.splitlines(): + parts = line[1:].split() + if len(parts) >= 2: + subs.add(os.path.normpath(os.path.join(tree.path, parts[1]))) + for line in sub.stdout.splitlines(): + if not line.strip() or line.startswith("-"): + continue + parts = line[1:].split() + if len(parts) < 2: + continue + spath = os.path.join(tree.path, parts[1]) + dirt = git_out(spath, "status", "--porcelain") + if dirt is None or dirt: + return (f"submodule {parts[1]} holds uncommitted work (a commit in the tree " + "cannot carry it) - commit it there first") + # ITS IGNORED WORK TOO: an `.env` in a submodule is archived by + # nobody - the tree's own archive lists the superproject's only. + extra = ignored_paths(spath) + if extra is None or extra: + return (f"submodule {parts[1]} holds ignored files no commit carries " + + (f"({extra[0]}{' and more' if len(extra) > 1 else ''})" if extra + else "(they could not be listed)") + + " - move them out first") + # ITS BRANCHES, TAGS AND STASH, NOT ONLY ITS HEAD (#170 item 5): a + # worktree's submodules keep their repositories under its own git + # directory (`.git/worktrees//modules/`), and a clone's under + # `.git/modules/` - either way the removal takes them. + why = unpublished_ref(spath, "--remotes", "every remote of it") + if why: + return (f"submodule {parts[1]}: {why} - push it there first; the tree's removal " + "takes the submodule's repository with it") + sstash = git(spath, "stash", "list") + if sstash.returncode != 0 or sstash.stdout.strip(): + return (f"submodule {parts[1]} has a stash (or its stash could not be read), which " + "the tree's removal would take with it") + # ITS HIDDEN EDITS TOO (#174, Copilot round 2): a file flagged + # skip-worktree or assume-unchanged in the submodule reads as clean + # there exactly as in the superproject (#170 A2). + shidden = hidden_edits(spath) + if shidden is None or shidden: + return (f"submodule {parts[1]}: " + ( + "its skip-worktree and assume-unchanged flags could not be read" if shidden is None + else f"{len(shidden)} file(s) flagged skip-worktree or assume-unchanged differ from " + f"its index (e.g. {shidden[0]})") + + " - git status hides them and no rescue would carry them") + # ITS REFLOGS TOO (#174, Copilot round 1): a commit made in the + # submodule and checked out away from is in no branch, tag or stash - + # only a reflog under its git directory names it - and the tree's + # removal takes that directory, which no loss plan of the tree reads. + why = unpublished_reflog(spath, "--remotes", "every remote of it") + if why: + return (f"submodule {parts[1]}: {why} - push it there first; the tree's removal " + "takes the submodule's repository with it") + # EVERY SUBMODULE REPOSITORY ITS GIT DIRECTORY KEEPS (#174, Copilot round + # 2), a DEINITIALIZED one's included: `git submodule status` shows it `-`, + # its working tree is gone, and its repository - branches, stash, reflog - + # stays under `/modules/`, which the removal takes. + gitdir = git_out(tree.path, "rev-parse", "--absolute-git-dir") + if not gitdir: + return "its git directory could not be read, so the submodule repositories it keeps are unknown" + why = module_repos_why(gitdir) + if why: + return why + "; the tree's removal takes that repository with it" + # A REPOSITORY NESTED ANYWHERE IN IT - a venv's `src/` (`pip install -e + # git+...`), node_modules and a cache directory included (#170 A3; #174, + # Copilot round 2) - and a directory that cannot be read keeps the tree. + nested, unread = nested_repos(tree.path, subs) + if nested: + return (f"a repository is nested inside it at {nested[0]} - its work is " + "no commit of this tree's; move it out first (a lane creates " + "worktrees, never clones)") + if unread: + return (f"{unread[0]} inside it could not be read, so whether a repository is nested " + "there is unknown") + return "" + + +def nested_repos(top: str, exclude: set = frozenset(), bare: bool = False) -> tuple: + """([every directory under `top` that holds a `.git`], [what could not + be read]) - `top` itself, `exclude`d paths (a tree's submodules) and + anything inside a nested repository are not entered. NOTHING ELSE IS + PASSED OVER (#174, Copilot round 2): this walk is the proof that a removal + takes no repository, and a checkout under a directory named + `__pycache__` or `.pytest_cache` is still a checkout. With `bare`, a + BARE repository (`is_bare_repository`) is found as one too.""" + found, unread = [], [] + stack = [top] + while stack: + d = stack.pop() + try: + with os.scandir(d) as it: + entries = list(it) + except FileNotFoundError: + continue + except OSError as exc: + unread.append(f"{d} ({exc.strerror or exc})") + continue + if d != top and (any(e.name == ".git" for e in entries) + or (bare and _bare_shaped(entries))): + found.append(d) + continue + for e in entries: + if e.name == ".git": continue - parts = line[1:].split() - if len(parts) < 2: + try: + if e.is_symlink() or not e.is_dir(follow_symlinks=False): + continue + except OSError: continue - spath = os.path.join(tree.path, parts[1]) - dirt = git_out(spath, "status", "--porcelain") - if dirt is None or dirt: - return (f"submodule {parts[1]} holds uncommitted work (a commit in the tree " - "cannot carry it) - commit it there first") - # ITS IGNORED WORK TOO: an `.env` in a submodule is archived by - # nobody - the tree's own archive lists the superproject's only. - extra = ignored_paths(spath) - if extra is None or extra: - return (f"submodule {parts[1]} holds ignored files no commit carries " - + (f"({extra[0]}{' and more' if len(extra) > 1 else ''})" if extra - else "(they could not be listed)") - + " - move them out first") - on = git(spath, "for-each-ref", "--count=1", "--contains", "HEAD", - "--format=%(refname)", "refs/remotes/") - if on.returncode != 0 or not on.stdout.strip(): - return (f"submodule {parts[1]} is at a commit no remote of it holds " - "- push it there first") - subs = set() - if sub.returncode == 0: - for line in sub.stdout.splitlines(): - parts = line[1:].split() - if len(parts) >= 2: - subs.add(os.path.normpath(os.path.join(tree.path, parts[1]))) - prune = set(CACHE_NAMES) | set(VENV_NAMES) - for base, dirs, files in os.walk(tree.path): - if base != tree.path and (".git" in dirs or ".git" in files): - if os.path.normpath(base) not in subs: - return (f"a repository is nested inside it at {base} - its work is " - "no commit of this tree's; move it out first (a lane creates " - "worktrees, never clones)") - dirs[:] = [d for d in dirs if d != ".git" and d not in prune - and os.path.normpath(os.path.join(base, d)) not in subs] + if os.path.normpath(e.path) not in exclude: + stack.append(e.path) + return sorted(found), unread + + +def _bare_shaped(entries: list) -> bool: + """A directory listing that is a git directory's own: a `HEAD` beside an + `objects/` - as `module_repos` reads one.""" + names = {e.name for e in entries} + return "HEAD" in names and "objects" in names + + +def is_bare_repository(path: str) -> bool: + """`path` is a git directory with no working tree of its own - a BARE + repository (`git init --bare`, a mirror, `.git`) - by its shape.""" + try: + with os.scandir(path) as it: + return _bare_shaped(list(it)) + except OSError: + return False + + +def repository_within(path: str) -> str: + """Why `path` holds a repository - it is one, or one lies anywhere inside + it - or that what is inside could not be read; else ''. What a directory + removal without a commit-level check (a cache, a sandbox) must never + take. A BARE repository is one (#174 review R2): it has no `.git`, and a + sandbox holding the only copy of a commit in one was removed as empty.""" + if os.path.lexists(os.path.join(path, ".git")) or is_bare_repository(path): + return f"it is a repository ({path})" + nested, unread = nested_repos(path, bare=True) + if nested: + return f"a repository lies inside it ({nested[0]})" + if unread: + return f"{unread[0]} inside it could not be read, so whether a repository lies there is unknown" + return "" + + +def lost_tips(path: str, cands: list, keep: list): + """The candidate objects at least one commit of whose history no ref or + object in `keep` (rev-list arguments: `--remotes=origin`, `--branches`, a + SHA) reaches - what a removal would take that is in no such place. A + candidate object that is gone (a reflog entry gc took) is already lost + and is not reported; one that is no commit cannot be proved kept and is. + + AN ANNOTATED TAG IS ITS OWN OBJECT (#174, Copilot round 1): a clone's tag + reached here as the tag object, was reported lost however it was kept, + and the self-check refused - and halted the sweep on - every clone with + one. A tag object is kept where `keep` names it by SHA (a verified + bundle's head, origin's own tag); its history is its peeled commit's, + asked like any other. None where git could not answer.""" + want = [] + for c in cands: + if c and c not in want and not re.match(r"^0+$", c): + want.append(c) + if not want: + return [] + kept_objects = [k for k in keep if SHA_RE.match(k)] + keep_refs = [k for k in keep if not SHA_RE.match(k)] + query = want + [f"{k}^{{commit}}" for k in kept_objects] + check = run(["git", "-C", path, "cat-file", "--batch-check=%(objectname) %(objecttype)"], + env=git_env(path), input_text="\n".join(query) + "\n") + lines = check.stdout.splitlines() + if check.returncode != 0 or len(lines) != len(query): + return None + commits, odd, tags = [], [], [] + for sha, line in zip(want, lines[:len(want)]): + parts = line.split() + otype = parts[-1] if parts else "" + if otype == "commit": + commits.append(sha) + elif otype == "tag": + tags.append(sha) + elif otype != "missing": + odd.append(sha) + # A SHA in `keep` counts through the commit it peels to: a tag object a + # bundle carries keeps that commit's history too. + keep_commits = [line.split()[0] for line in lines[len(want):] + if len(line.split()) == 2 and line.split()[1] == "commit"] + if tags: + peel = run(["git", "-C", path, "cat-file", "--batch-check=%(objectname) %(objecttype)"], + env=git_env(path), input_text="".join(f"{t}^{{commit}}\n" for t in tags)) + peeled = peel.stdout.splitlines() + if peel.returncode != 0 or len(peeled) != len(tags): + return None + for tag, line in zip(tags, peeled): + parts = line.split() + if tag not in kept_objects: + odd.append(tag) + if len(parts) == 2 and parts[1] == "commit" and parts[0] not in commits: + commits.append(parts[0]) + if not commits: + return odd + proc = run(["git", "-C", path, "rev-list", *commits, "--not", *keep_refs, *keep_commits], + env=git_env(path), timeout=GIT_TIMEOUT) + if proc.returncode != 0: + return None + out = set(proc.stdout.split()) + return [c for c in commits if c in out] + odd + + +def origin_tag_objects(path: str, tags: list) -> list: + """The annotated tag objects among `tags` origin holds AS THEY ARE (its + `ls-remote --tags` names that very object) - kept by origin, so not the + only copy. Where origin cannot be asked, none: such a tag is bundled.""" + if not tags: + return [] + proc = git(path, "ls-remote", "--tags", "origin", timeout=GIT_NET_TIMEOUT) + if proc.returncode != 0: + return [] + have = {line.split("\t", 1)[0] for line in proc.stdout.splitlines() if "\t" in line} + return [t for t in tags if t in have] + + +def unpublished_ref(path: str, remotes: str, where: str) -> str: + """'' when every branch, tag and HEAD of the repository at `path` is in a + remote-tracking ref `remotes` names (`--remotes=origin`, `--remotes`); + else which one is not, or that it could not be checked - both keep.""" + proc = git(path, "for-each-ref", "--format=%(refname)%09%(objecttype)%09%(objectname)" + "%09%(*objecttype)%09%(*objectname)", "refs/heads/", "refs/tags/") + if proc.returncode != 0: + return "its branches and tags could not be read" + tips: dict = {} + for line in proc.stdout.splitlines(): + cols = line.split("\t") + if len(cols) < 5: + continue + ref, otype, sha, ptype, psha = cols[:5] + commit = sha if otype == "commit" else (psha if ptype == "commit" else "") + if not commit: + return (f"its tag {ref[len('refs/tags/'):]} names no commit, so whether {where} " + "holds it cannot be checked") + tips.setdefault(commit, ref) + head = git_out(path, "rev-parse", "-q", "--verify", "HEAD^{commit}") + if head: + tips.setdefault(head, "HEAD") + lost = lost_tips(path, list(tips), [remotes]) + if lost is None: + return f"whether {where} holds its branches and tags could not be read" + if not lost: + return "" + ref = tips.get(lost[0], lost[0]) + for prefix, kind in (("refs/heads/", "branch "), ("refs/tags/", "tag ")): + if ref.startswith(prefix): + return f"its {kind}{ref[len(prefix):]} holds commits {where} lacks" + return f"its {ref} holds commits {where} lacks" + + +def unpublished_reflog(path: str, remotes: str, where: str) -> str: + """'' when every commit a reflog of the repository at `path` names is in + a remote-tracking ref `remotes` names; else how many are not, or that + the reflogs could not be read - both keep.""" + gitdir = git_out(path, "rev-parse", "--absolute-git-dir") + if not gitdir: + return f"its git directory could not be read, so whether {where} holds what its reflogs name is unknown" + files, why = reflog_files(gitdir) + if files is None: + return why + shas, why = reflog_shas(files) + if shas is None: + return why + lost = lost_tips(path, shas, [remotes]) + if lost is None: + return f"whether {where} holds the commits its reflogs name could not be read" + if lost: + return (f"its reflog names {len(lost)} commit(s) {where} lacks (e.g. {short(lost[0])}), " + "in no branch, tag or stash") return "" +def hidden_edits(path: str): + """Files flagged skip-worktree or assume-unchanged whose content in the + tree differs from the index - edits `git status` does not show and `git + add -A` does not take (#170 A2). A flagged file that is not there (a + sparse checkout's) is no edit. None where git could not answer.""" + proc = git(path, "ls-files", "-v", "-s", "-z") + if proc.returncode != 0: + return None + files, out = [], [] + for rec in proc.stdout.split("\0"): + meta, sep, rel = rec.partition("\t") + cols = meta.split() + if not sep or len(cols) < 4: + continue + tag, mode, sha = cols[0], cols[1], cols[2] + if not (tag == "S" or tag.islower()): + continue + full = os.path.join(path, rel) + if not os.path.lexists(full) or mode == "160000": + continue + if mode == "120000": + blob = git(path, "cat-file", "blob", sha) + try: + same = blob.returncode == 0 and os.readlink(full) == blob.stdout + except OSError: + same = False + if not same: + out.append(rel) + continue + files.append((rel, sha)) + if files: + proc = run(["git", "-C", path, "hash-object", "--stdin-paths"], env=git_env(), + input_text="".join(rel + "\n" for rel, _ in files)) + got = proc.stdout.split() + if proc.returncode != 0 or len(got) != len(files): + return None + out += [rel for (rel, sha), now in zip(files, got) if now != sha] + return sorted(out) + + def ignored_work(tree: Tree): """Ignored paths that are NOT caches: a commit cannot carry them, so they are archived beside the bundle before the tree goes. None where git could @@ -1268,14 +1863,19 @@ def ignored_paths(path: str): rel = rel.rstrip("/") if not rel: continue - parts = rel.split("/") - if any(p in CACHE_NAMES for p in parts): + # THE ENTRY'S OWN NAME DECIDES, never an ancestor's (#170 A6): with + # `--directory` an ignored cache or build directory is listed as + # itself, so a path BENEATH a directory named `build` is an ignored + # file in a tracked directory - `docker/build/prod.env` - and was + # dropped unarchived while `top.env` beside it was archived. + name = os.path.basename(rel) + if name in CACHE_NAMES: continue full = os.path.join(path, rel) - if os.path.isdir(full) and (os.path.basename(rel) in VENV_NAMES + if os.path.isdir(full) and (name in VENV_NAMES or os.path.exists(os.path.join(full, "pyvenv.cfg"))): continue - if any(p in BUILD_NAMES or p.endswith(".egg-info") for p in parts): + if name in BUILD_NAMES or name.endswith(".egg-info"): continue if rel.endswith((".pyc", ".pyo")): continue @@ -1300,6 +1900,7 @@ class Options: self.live = list(ns.live or []) self.fetch = bool(ns.fetch) or self.yes self.estate = ns.estate or "" + self.push_untracked = bool(ns.push_untracked) class Item: @@ -1338,11 +1939,19 @@ class Sweep: self.archive = "" self.failed = 0 self.deps: dict = {} + self.deps_partial: list = [] + self.walk_lane_states: list = [] + self.shape_repos: list = [] self.estate_root = "" self.fetched: dict = {} self.rescues: list = [] self.dispositions: list = [] self.acting = False + self.verified: dict = {} # common dir -> SHAs a push of this run put on origin + self.bundled: dict = {} # common dir -> heads of this run's verified bundles + self.halt = "" # why no further act is taken + self.refused_check = "" # the self-check's refusal: exit 2 + self.usage = "" # a usage error found once the trees are known: 64 # ------------------------------------------------------------- reading def gather(self) -> None: @@ -1350,6 +1959,18 @@ class Sweep: self.refusal = ("no lanes-edit.sh beside this command or on PATH: which trees " "are this lane's is #97's inventory, and nothing else answers it") return + # UNDER --yes THE REGISTER IS FETCHED FIRST, AND A FETCH THAT FAILED IS A + # REFUSAL (#170 A1): a binding read from a ref as it stood here is not + # where the lane is bound now, and from outside its binding a lane is + # UNKNOWN, never dead (Amendment 18(b)). An inherited LANES_NO_FETCH=1 + # does not skip it. + if self.opts.yes: + why = self.helper.fetch_register() + if why: + self.refusal = ("--yes acts only on the register as origin holds it NOW, and it " + f"could not be fetched: {why}. Where the lane is bound is then " + "unknown, never 'here' (Amendment 18(b)); nothing was changed") + return # THE CANONICAL SPELLING BEFORE ANY PATH IS DERIVED FROM IT: scratch, # caches, rescue names and the Lane: trailer comparison all use it. canon, why = self.helper.canon_lane(self.lane) @@ -1382,16 +2003,35 @@ class Sweep: # right place would do; it is never fetched or touched from here. self._lane_gate() self.procs = ProcScan() + if self.procs.unknown: + self.notes.append( + f"{len(self.procs.unknown)} process(es) of this account could not be read in " + "/proc (" + ", ".join(f"{pid} {comm}" for pid, comm, _h, _a in + self.procs.unknown[:6]) + + (" ..." if len(self.procs.unknown) > 6 else "") + "): a tree their command " + "line or parent's directory places them in is of unknown liveness and kept") self._discover() + self.usage = self._live_usage() + if self.usage: + return + # THE ESTATE'S CHECKOUTS BY ITS SHAPE (a quick walk): beside each one + # another lane's control root may sit. + self.shape_repos = estate_repos(self.estate_root) if self.estate_root else [] # WHO ELSE CLAIMS THESE TREES is read before anything is written: a # claim that cannot be read refuses, and a refusal never fetches. self._other_lanes() self._lane_claims() if self.opts.fetch and not self.refusal: self._fetch_all() - self.deps = dependents_map(estate_repos(self.estate_root)) if self.estate_root else {} for tree in self.trees: read_tree(tree) + self._dependents() + # EVERY CONTROL ROOT THE WHOLE-ESTATE WALK FOUND IS A PLACE ANOTHER LANE'S + # CLAIM MAY BE (#174, Copilot round 2): `host/vendor/.lane-state`, with + # `host` a repository and no log naming it, was found by that walk and + # thrown away - and a clone another lane claims from there was removed. + if self.walk_lane_states: + self._other_lanes(self.walk_lane_states) for tree in self.trees: self._liveness(tree) if tree.is_git and tree.exists: @@ -1401,19 +2041,63 @@ class Sweep: tree.merge = self.evidence.judge(tree.path, slug_of(tree.origin_url), tree.branch, tree.head) + def _dependents(self) -> None: + """What leans on the trees. WHERE A REMOVAL WOULD DELETE A DIRECTORY - + a clone the table could take, or `--include-scratch` - the whole + estate is walked (#170 A11): every directory but caches, so a clone + in a plain directory inside a repository is seen, and what the walk + could not read makes the scan PARTIAL, which removes no such + directory. A worktree's removal deletes no objects (they are its + checkout's), so where only worktrees are candidates the estate's + shape is enough: on Eagle the whole walk is 683,000 directories.""" + whole = self.opts.scratch or any( + t.exists and t.kind == "clone" + and (t.owner != "foreign" or (self.opts.foreign and t.foreign_kind != "other-lane")) + for t in self.trees) + unread: list = [] + if whole: + walk = walk_estate(self.estate_root) + unread += walk.unread + self.walk_lane_states = walk.lane_states + self.deps = dependents_map(walk.repos, unread) + self.deps_partial = unread + else: + self.deps = dependents_map(self.shape_repos, unread) + self.deps_partial = [] + + def _live_usage(self) -> str: + """`--live` names writers' trees, so each path must name one of the + table's (#170 B4): a misspelt path was accepted in silence, and the + writer it meant to protect was not protected. `none` stands alone.""" + named = [n for n in self.opts.live if n != "none"] + if named and len(named) != len(self.opts.live): + return "--live none says no writer is live; it takes no path beside it" + for n in named: + if not any(under(t.path, n) or under(n, t.path) for t in self.trees): + return (f"--live {n} names no tree of this sweep (misspelt?); the trees are " + "the lane's inventory, its roots and its checkout's registrations") + return "" + def _lane_claims(self) -> None: """A tree #97's inventory does not name is still THIS LANE'S when it stands under the lane's own worktree root (`.lane-worktrees//`, which only this lane's starts make), or sits in the checkout's - `.claude/worktrees` on a branch whose OWN commits - those its origin's - default branch lacks - carry this lane's `Lane:` trailer. Another - lane's claim still wins. Without this a lane whose trees were never - recorded read as "nothing to retire", and #163's gate passed it (B1). + `.claude/worktrees` with its HEAD - a commit its origin's default + branch lacks - carrying this lane's `Lane:` trailer as its own. + Another lane's claim still wins. Without this a lane whose trees were + never recorded read as "nothing to retire", and #163's gate passed it + (B1). AND A LANE WITH NO SNAPSHOT AT ALL (#97's state NONE) BUT TREES OF ITS OWN IS REFUSED: which of them are its is recorded nowhere, so the gate does not pass it until a handoff records them or a person retires - them.""" + them. + + HEAD'S OWN TRAILER, NOT ANY OF THE BRANCH'S (#174 review R1): a tree + another lane STACKED on this lane's commit carries this lane's trailer + below its tip, and adopting it on that removed the other lane's tree + and pushed its in-progress edit to `rescue//...` - and two + lanes sharing one checkout both adopted it.""" roots = self._roots() lane_root = roots[-1] if roots else "" claude_root = roots[0] if self.facts.dir and roots else "" @@ -1426,7 +2110,8 @@ class Sweep: if lane_root and under(t.path, lane_root) and t.real != real(lane_root): t.owner = "lane-root" elif claude_root and t.exists and under(t.path, claude_root) \ - and self.lane.lower() in own_trailers(t.path): + and self.lane.lower() in own_trailers(t.path) \ + and head_lane_trailer(t.path).lower() == self.lane.lower(): t.owner = "trailer" claimed = [t for t in self.trees if t.owner in ("lane-root", "trailer")] if self.facts.state == "NONE" and claimed: @@ -1475,6 +2160,11 @@ class Sweep: regs_by_real: dict = {} for co in checkouts: regs = worktree_registrations(co) + if regs is None: + self.refusal = self.refusal or ( + f"the worktree registrations of {co} could not be read (git worktree list " + "failed), so which trees it holds is unknown (#170 G5, Amendment 7(d))") + continue for i, reg in enumerate(regs): if i == 0 or reg.get("bare"): continue # the main worktree is the checkout itself @@ -1487,10 +2177,19 @@ class Sweep: t.registered = True t.checkout = t.checkout or co t.locked = bool(reg.get("locked")) + t.reg = reg for root in self._roots(): try: names = sorted(os.listdir(root)) - except OSError: + except (FileNotFoundError, NotADirectoryError): + continue + except OSError as exc: + # A ROOT THAT CANNOT BE LISTED IS NOT AN ABSENT ROOT (#170 G1): read + # as one, an empty inventory let the dry run exit 0 and clear #163's + # gate over trees nobody could see. + self.refusal = self.refusal or ( + f"the lane's worktree root {root} could not be listed " + f"({exc.strerror or exc}), so which trees stand in it is unknown") continue for name in names: full = os.path.join(root, name) @@ -1504,8 +2203,24 @@ class Sweep: seen[real(full)] = Tree(full, "foreign", kind) self.trees = sorted(seen.values(), key=lambda t: (t.owner != "inventory", t.path)) - def _other_lanes(self) -> None: - """A FOREIGN tree another lane's inventory names is THAT lane's.""" + def _other_lanes(self, extra: list = ()) -> None: + """A FOREIGN tree another lane's inventory names is THAT lane's. + + EVERY LANE'S CONTROL ROOT IS READ, wherever #97 keeps it (#170 G6): a + lane whose checkout is nested (`/group/repo`) keeps its + `.lane-state` beside that checkout, not beside this lane's, so the + parents read are #97's own rungs for every lane - the override, the + parent of every lane's recorded `dir` in the register, `$PROJECTS_ROOT` + - beside this lane's own and beside every checkout of the estate. A + register that cannot be read refuses: whose claims were missed would + be unknown. + + ONLY THIS SWEEP'S ESTATE, AND NEVER A REFUSAL WITH NO TREE (#174 review + R3): the register names every lane on the workstation, so one pathless + or unreadable sidecar in ANY estate made the sweep exit 2 for a lane + with nothing on disk - every lane-end blocked. A recorded `dir` outside + the estate root is not read, and with no tree a claim can take away + nothing, so one that cannot be read is a note.""" parents = set() if self.facts.root: parents.add(os.path.dirname(self.facts.root.rstrip("/"))) @@ -1513,14 +2228,40 @@ class Sweep: parents.add(os.path.join(os.path.dirname(self.facts.dir.rstrip("/")), ".lane-state")) if os.environ.get("LANES_LANE_STATE_ROOT"): parents.add(os.environ["LANES_LANE_STATE_ROOT"]) + if os.environ.get("PROJECTS_ROOT"): + parents.add(os.path.join(os.environ["PROJECTS_ROOT"], ".lane-state")) + parents.update(os.path.join(os.path.dirname(r.rstrip("/")), ".lane-state") + for r in self.shape_repos) + parents.update(extra) claimed: dict = {} def unreadable(what: str) -> None: - # A CLAIM THAT CANNOT BE READ IS NOT "NO CLAIM" (Amendment 7(d)). + # A CLAIM THAT CANNOT BE READ IS NOT "NO CLAIM" (Amendment 7(d)) - + # but a claim only takes a tree from this lane, so with no tree it + # can lower nothing: a note then, never a refusal (#174 review R3). + if not self.trees: + text = (f"{what} could not be read; this lane has no tree for another lane's " + "claim to take, so the sweep goes on - read it, or move it aside if it " + "is debris") + if text not in self.notes: + self.notes.append(text) + return self.refusal = self.refusal or ( f"{what} could not be read, so whether another lane claims these trees is " "unknown; read it, or move it aside if it is debris") + dirs, why = register_lane_dirs(self.helper) + if why: + unreadable(f"every lane's recorded directory ({why})") + for other, odir in dirs.items(): + if other.lower() == self.lane.lower(): + continue + parent = os.path.join(os.path.dirname(odir.rstrip("/")), ".lane-state") + # THIS SWEEP'S ESTATE ONLY (#174 review R3): a lane recorded in + # another estate keeps its claims there. + if self.estate_root and not under(parent, self.estate_root): + continue + parents.add(parent) for parent in sorted(parents): try: lanes = os.listdir(parent) @@ -1530,7 +2271,7 @@ class Sweep: unreadable(f"the lane-state directory {parent}") continue for other in lanes: - if other == self.lane: + if other.lower() == self.lane.lower(): continue tdir = os.path.join(parent, other, "trees") try: @@ -1597,7 +2338,16 @@ class Sweep: if not git_out(probe, "config", "--get", "remote.origin.url"): self.fetched[common] = "no origin remote" continue - proc = git(probe, "fetch", "--prune", "--quiet", "origin", timeout=GIT_NET_TIMEOUT) + # THE REFSPEC IS PINNED, as `status --fetch` pins it (#170 item 6): + # a mirror-style `remote.origin.fetch` (`+refs/heads/*:refs/heads/*`) + # would let `--prune` delete local-only branches before they were + # classified or rescued. `--refmap` keeps the configured refspec from + # mapping anything, and neither `fetch.pruneTags` nor the remote's + # `pruneTags` may take a local tag with it. + proc = git(probe, "-c", "fetch.pruneTags=false", "-c", "remote.origin.pruneTags=false", + "fetch", "--prune", "--quiet", "--recurse-submodules=no", + "--refmap=+refs/heads/*:refs/remotes/origin/*", "origin", + "+refs/heads/*:refs/remotes/origin/*", timeout=GIT_NET_TIMEOUT) self.fetched[common] = "ok" if proc.returncode == 0 else ( "fetch failed: " + _first_line(proc.stderr)) @@ -1621,6 +2371,10 @@ class Sweep: pid, comm, kind, where = holders[0] tree.live = (True, f"process {pid} ({comm or '?'}) has its {kind} there") return + hidden = self.procs.unknown_for(tree.path) + if hidden: + tree.live = (None, "liveness could not be read: " + hidden) + return elif self.procs is not None: tree.live = (None, "liveness could not be read: " + self.procs.why) return @@ -1673,6 +2427,17 @@ class Sweep: "(i)). State the count: --live for each live writer, or " "--live none") return + # A TREE THIS SESSION RECORDED IS ONE ITS OWN WRITERS MAY STAND IN (#170 + # B5): their liveness is never asked of the transcript - it is this + # one - so the coordinator's count is wanted whatever the holder reads. + mine = [r for r in f.inventory if self.caller and r.get("writer") == self.caller] + if self.opts.yes and mine and not self.opts.live: + self.refusal = (f"this session recorded {len(mine)} of lane {self.lane}'s trees (e.g. " + f"{mine[0].get('path') or '?'}), so its own writers may be in them, " + "and only its coordinator knows which are live (ListAgents; " + "Amendment 17 Addendum 1 (i)). State the count: --live " + "for each live writer, or --live none") + return if f.verdict == "indeterminate": self.refusal = (f"#97's reconciliation of lane {self.lane} is INDETERMINATE: " + (f.verdict_why or "a read it rests on failed")) @@ -1726,6 +2491,31 @@ class Sweep: if t.locked: put("keep", "its directory is gone and the registration is LOCKED; " f"git -C {t.checkout} worktree unlock {t.path} first") + return + # ITS GIT DIRECTORY MAY KEEP SUBMODULE REPOSITORIES (#174, Copilot + # round 2): `/modules/`, with branches, a stash or a + # reflog no remote holds, goes with the registration. + admin = worktree_admin(t.checkout, t.path) if t.checkout else "" + mod_why = module_repos_why(admin) if admin else "" + if mod_why: + put("keep", "its directory is gone, and " + mod_why + "; pruning the " + "registration would take that repository with it") + return + # THE REGISTRATION MAY BE THE ONLY POINTER TO ITS COMMITS (#170 + # A4): a detached HEAD, or a reflog entry, that no branch, tag or + # origin holds is lost with the registration's git directory. + try: + lost = self._lost(t) + except _ActFailed as exc: + put("keep", "its directory is gone, and " + str(exc).replace( + "; left in place", "")) + return + if lost: + put("rescue+prune", f"the directory is gone and git still registers it; its " + f"registration names {len(lost)} commit(s) no branch, tag " + "or origin holds (its HEAD or reflog): rescued - a rescue " + "branch pushed where there is an origin, and a bundle - " + "then that one registration pruned") else: put("prune", "the directory is gone and git still registers it") else: @@ -1753,6 +2543,14 @@ class Sweep: + "; remedy: in each dependent `git repack -a -d` then drop its " "objects/info/alternates, or re-point the local remote - then sweep again") return + if t.kind == "clone": + partial = self._deps_partial(t.path) + if partial: + put("keep", "it is a CLONE, removed by deleting its directory, and the search for " + "repositories that lean on it (alternates, a local remote) was " + "partial: " + _first_few(partial) + + " - a dependent nobody saw would lose its objects") + return if self.opts.fetch: state = self.fetched.get(real(t.common), "") if state and state not in ("ok", "no origin remote"): @@ -1767,8 +2565,20 @@ class Sweep: return dirty = len(t.status_lines) if dirty: + untracked = sum(1 for ln in t.status_lines if ln.startswith("?? ")) + if untracked and not self.opts.push_untracked: + files = untracked_files(t.path) + why = (lfs_untracked_why(t.path, files) if files is not None else + "its untracked files could not be listed, so whether git-lfs content " + "is among them is unknown" if lfs_configured(t.path) else "") + if why: + put("keep", why) + return put("wip-rescue+remove", f"{dirty} dirty or untracked path(s): a WIP commit on " - "a rescue branch, pushed, then removed") + "a rescue branch, pushed, then removed" + + (f"; its {untracked} untracked path(s) go to the bundle only, never pushed " + "(--push-untracked pushes them)" if untracked and not self.opts.push_untracked + else "")) return if t.unborn: put("remove", "an unborn branch with nothing in it") @@ -1801,6 +2611,32 @@ class Sweep: put("rescue+remove", f"{t.ahead or 'unpublished'} commit(s) not on origin and {why}: " "a rescue branch, pushed, then removed") return + # A LANE'S UNREVIEWED COMMITS NEVER GO ONTO SOMEBODY'S BRANCH AS IS + # (#170 A7): not onto one an open pull request names (a teammate's + # under review), not onto one whose tip carries another lane's + # `Lane:` trailer, and not where whether a PR names it could not be + # read. They go to a rescue branch of this lane's instead. + if t.remote_tip: + other = t.remote_trailer if (t.remote_trailer and + t.remote_trailer.lower() != self.lane.lower()) else "" + # GH'S OPEN WINS HERE WHATEVER THE REGISTER SAYS (#174, Copilot round + # 2): a LANDED line naming a PR gh still answers OPEN made the verdict + # "moved", and the lane's commits were pushed onto that open branch. + gh_open = m.get("gh_open") or [] + why = (m["why"] + ", and a lane's unreviewed commits never go onto an open pull " + "request's branch" if m.get("state") == "open" else + f"gh still answers PR #{gh_open[0]} OPEN for it, whatever the register's " + "LANDED line says, and a lane's unreviewed commits never go onto an open pull " + "request's branch" if gh_open else + f"origin/{t.remote_branch} carries lane {other}'s work (its tip's Lane: " + "trailer)" if other else + f"whether an open pull request names origin/{t.remote_branch} is unknown " + f"({m.get('why', 'not read')})" if m.get("state") == "unknown" else "") + if why: + put("rescue+remove", f"{t.ahead} commit(s) not on origin/{t.remote_branch}, and " + f"{why}: a rescue branch, pushed, then removed - " + f"origin/{t.remote_branch} is left as it is") + return put("push+remove", (f"{t.ahead} commit(s) not on origin/{t.remote_branch}" if t.remote_tip else f"origin has no {t.remote_branch}") + ": pushed as is, then removed") @@ -1842,7 +2678,14 @@ class Sweep: continue # a checkout is a tree, and the table above has it holders = self.procs.holders(full) if self.procs and self.procs.available else [] lean = leaned_on(full, "", self.deps) - if lean: + inner = self._trees_inside(full) + if inner: + # TREES UNDER IT ARE THE TABLE'S FIRST (#170 item 7): an archive + # and an rmtree of the scratch would take a worktree nested in it + # - FOREIGN, live or the lane's own - with its git directory. + it = Item("scratch", full, "keep", f"{inner}: the tree table retires what is " + "under it first, and the scratch goes on a later sweep", True) + elif lean: it = Item("scratch", full, "load-bearing", "LOAD-BEARING: " + _lean_text(lean) + "; it is never removed until they are re-pointed", True) elif holders: @@ -1852,6 +2695,13 @@ class Sweep: elif self.procs is not None and not self.procs.available: it = Item("scratch", full, "keep", "liveness could not be read: " + self.procs.why, True) + elif self.procs is not None and self.procs.unknown_for(full): + it = Item("scratch", full, "keep", "liveness could not be read: " + + self.procs.unknown_for(full), True) + elif self._deps_partial(full): + it = Item("scratch", full, "keep", "it is removed by deleting its directory, and " + "the search for repositories that lean on it was partial: " + + _first_few(self._deps_partial(full)), True) else: it = Item("scratch", full, "archive+remove", "lane scratch: tar and sha256 under the sweeps directory, then removed", @@ -1887,10 +2737,23 @@ class Sweep: owner = self._cache_owner(path, roots, trees) # LIVENESS IS THE OWNER'S, not the cache's: a writer stands in # the tree, not in its __pycache__. - if self.procs is None or not self.procs.available or self.procs.holders(owner): + if self.procs is None or not self.procs.available or self.procs.holders(owner) \ + or self.procs.unknown_for(owner): continue if tracked(path) is not False: continue # tracked, or its tracking is unknown + # A CACHE THAT HOLDS A REPOSITORY HOLDS WORK (#174, Copilot round + # 1): `pip install -e git+...` clones into `/src/`, and + # the tree kept for that clone (#170 A3) still gave up its venv - + # the clone and its unpushed fix with it. + inner = repository_within(path) + if inner: + it = Item("cache", path, "keep", f"{inner}: its work is no cache's, and the " + "directory is never removed while it is there", False) + it.size = size + it.extra["owner"] = owner + self.items.append(it) + continue it = Item("cache", path, "remove", "a dependency cache or bytecode: it holds no " "work and is rebuilt", False) it.size = size @@ -1902,6 +2765,29 @@ class Sweep: return (t.disposition in ("foreign", "live", "load-bearing") or t.live is not None or t.contested or t.result.startswith(("FAILED", "left"))) + def _trees_inside(self, path: str) -> str: + """Why `path` holds trees, or '': a tree of the table under it, or + any `.git` beneath it (or a directory there that cannot be read).""" + for t in self.trees: + if t.real != real(path) and under(t.path, path): + return f"a tree of this table lies under it ({t.path})" + nested, unread = nested_repos(path) + if nested: + return f"a repository lies under it ({nested[0]})" + if unread: + return f"{unread[0]} under it could not be read, so whether a tree lies there is unknown" + return "" + + def _deps_partial(self, path: str) -> list: + """Why the search for what leans on `path` is not complete: what the + estate walk could not read, and `path` lying outside the estate root + it searched.""" + out = list(self.deps_partial) + if self.estate_root and not under(path, self.estate_root): + out.append(f"{path} lies outside the estate root {self.estate_root}, the only place " + "dependents were looked for") + return out + @staticmethod def _cache_owner(path: str, roots: list, trees: list) -> str: """The tree a cache is in, else the top directory under the lane's @@ -1995,8 +2881,23 @@ class Sweep: for t in self.trees: if t.disposition in ("live", "foreign", "keep", "gone", "load-bearing", ""): continue + if self.halt: + # NOTHING MORE IS ACTED ON once a self-check refused or the + # register refused a line (#170 B6): what is left is exactly + # as the table found it. + t.result = "left: " + self.halt + continue try: self._act_tree(t) + except _Keep as exc: + t.disposition = "keep" + t.result = "left: " + str(exc) + except _SelfCheck as exc: + t.result = "REFUSED by the self-check: " + str(exc) + self.refused_check = f"{t.path}: {exc}" + self.halt = (f"the self-check refused {t.path} ({exc}); no further act is taken " + "until that is understood") + note(f"REFUSED by the self-check before removing {t.path}: {exc}") except _ActFailed as exc: t.result = "FAILED: " + str(exc) self.failed += 1 @@ -2005,8 +2906,20 @@ class Sweep: self.failed += 1 self._record(t) for i in self.items: + if self.halt: + if i.retire or i.disposition in ("remove", "archive+remove", "delete", + "delete+remote"): + i.result = "left: " + self.halt + continue try: self._act_item(i) + except _SelfCheck as exc: + i.result = "REFUSED by the self-check: " + str(exc) + self.refused_check = f"{i.path}: {exc}" + self.halt = (f"the self-check refused {i.kind} {i.path} ({exc}); no further act " + "is taken until that is understood") + note(f"REFUSED by the self-check before deleting {i.kind} {i.path}: {exc}") + self._archive_dir() except (_ActFailed, OSError) as exc: i.result = "FAILED: " + str(exc) self.failed += 1 @@ -2031,10 +2944,10 @@ class Sweep: self.archive = path return self.archive - def _rescue_name(self, t: Tree) -> str: - base = f"rescue/{safe_ref_part(self.lane)}/{t.slice}-{self.stamp}" + def _rescue_name(self, t: Tree, suffix: str = "") -> str: + base = f"rescue/{safe_ref_part(self.lane)}/{t.slice}-{self.stamp}{suffix}" name, n = base, 1 - while git(t.path, "rev-parse", "-q", "--verify", f"refs/heads/{name}").returncode == 0: + while git(t.gitpath, "rev-parse", "-q", "--verify", f"refs/heads/{name}").returncode == 0: n += 1 name = f"{base}-{n}" return name @@ -2050,6 +2963,9 @@ class Sweep: if held: pid, comm, kind, _w = held[0] return f"process {pid} ({comm or '?'}) has its {kind} there" + hidden = fresh.unknown_for(path) + if hidden: + return "liveness could not be read: " + hidden if writer and UUID_RE.match(writer) and writer != self.caller: answer = self.helper.transcript_live(writer, fresh=True) if answer != "none": @@ -2057,6 +2973,152 @@ class Sweep: + ("live" if answer == "live" else "of unknown liveness")) return "" + # ---------------------------------------------- what a removal would take + def _common(self, path: str) -> str: + common = git_out(path, "rev-parse", "--git-common-dir") or "" + if common and not os.path.isabs(common): + common = os.path.normpath(os.path.join(path, common)) + return real(common) if common else "" + + def _loss_plan(self, t: Tree): + """(where git is asked, candidate commits, what keeps them) for the + removal `t`'s disposition makes - or why it could not be read. + + A WORKTREE'S removal takes its own git directory: its HEAD and that + HEAD's reflog (and, where its branch is deleted after it, the branch + and the branch's reflog). Every branch, tag and stash of the + repository stays. A PRUNED registration takes the same, read from its + git directory, the working tree being gone already. A CLONE's removal + takes everything in it: every ref but its remote-tracking ones, and + every reflog.""" + if not t.exists: + co = t.checkout + admin = worktree_admin(co, t.path) if co else "" + if not admin: + return None, f"the git directory of its registration in {co or '?'} was not found" + logs, why = reflog_shas([os.path.join(admin, "logs", "HEAD")]) + if logs is None: + return None, why + return (co, [(t.reg or {}).get("head", "")] + logs, + ["--remotes=origin", "--branches", "--tags", "--glob=refs/stash"]), "" + gitdir = git_out(t.path, "rev-parse", "--absolute-git-dir") + if not gitdir: + return None, "its git directory could not be read" + if t.kind == "clone": + refs = git(t.path, "for-each-ref", "--format=%(refname) %(objectname) %(objecttype)", + "refs/") + if refs.returncode != 0: + return None, "its refs could not be read" + rows = [line.split(" ") for line in refs.stdout.splitlines() + if len(line.split(" ")) == 3 and not line.startswith("refs/remotes/")] + tips = [sha for _ref, sha, _otype in rows] + tags = [sha for ref, sha, otype in rows if ref.startswith("refs/tags/") and otype == "tag"] + files, why = reflog_files(gitdir) + if files is None: + return None, why + logs, why = reflog_shas(files) + if logs is None: + return None, why + return (t.path, tips + [t.head] + logs, + ["--remotes=origin"] + origin_tag_objects(t.path, tags)), "" + logs_files = [os.path.join(gitdir, "logs", "HEAD")] + keep = ["--remotes=origin"] + if t.disposition == "remove+delete-branch" and t.branch: + logs_files.append(os.path.join(t.common, "logs", "refs", "heads", *t.branch.split("/"))) + keep.append(f"--exclude={t.branch}") + keep += ["--branches", "--tags", "--glob=refs/stash"] + logs, why = reflog_shas(logs_files) + if logs is None: + return None, why + return (t.path, [t.head] + logs, keep), "" + + def _lost(self, t: Tree) -> list: + """The commits the removal would take that are on origin, in a + surviving ref, pushed and seen this run, or in a bundle of this run - + none of them. Raises when that cannot be read: unknown is never + 'nothing'.""" + plan, why = self._loss_plan(t) + if plan is None: + raise _ActFailed("which commits its removal would take could not be read (" + why + + "); left in place" + (f"; its rescue is at {t.rescue}" + if t.rescue else "")) + where, cands, keep = plan + common = self._common(where) + kept = sorted(self.verified.get(common, set()) | self.bundled.get(common, set())) + lost = lost_tips(where, cands, keep + kept) + if lost is None: + raise _ActFailed("whether its commits are kept elsewhere could not be read (git " + "rev-list failed); left in place") + return lost + + def _branch_lost(self, co: str, branch: str, tip: str) -> list: + """What deleting `refs/heads/` (at `tip`) in `co` would take + that no origin ref, other branch, tag, stash, push or bundle of this + run keeps: its tip and every commit its reflog names. Raises when + that cannot be read - unknown is never 'nothing'.""" + common = self._common(co) + if not common: + raise _ActFailed("the repository's git directory could not be read; the branch is left") + logs, why = reflog_shas([os.path.join(common, "logs", "refs", "heads", *branch.split("/"))]) + if logs is None: + raise _ActFailed(why + "; the branch is left") + keep = ["--remotes=origin", f"--exclude={branch}", "--branches", "--tags", + "--glob=refs/stash"] + kept = sorted(self.verified.get(common, set()) | self.bundled.get(common, set())) + lost = lost_tips(co, [tip] + logs, keep + kept) + if lost is None: + raise _ActFailed("whether its commits are kept elsewhere could not be read (git " + "rev-list failed); the branch is left") + return lost + + def _self_check(self, t: Tree) -> None: + """NOTHING IS DELETED THAT IS NOT FIRST ON ORIGIN OR IN A BUNDLE, asked + one last time of git itself, right before the removal: every commit + the removal would take must be reachable from origin, from a ref that + stays, from a push seen on origin this run, or from the heads of a + bundle this run wrote and verified. Anything else is a defect in the + table above, and the sweep stops (exit 2) rather than act on it.""" + try: + lost = self._lost(t) + except _ActFailed as exc: + raise _SelfCheck(str(exc)) + if lost: + raise _SelfCheck(f"{len(lost)} commit(s) it would take are neither on origin nor in a " + f"bundle (e.g. {short(lost[0])})") + + def _bundle_lost(self, t: Tree, lost: list, label: str) -> str: + """A bundle of commits no ref names: each is given a temporary ref + (`refs/lane-worktrees/-/