diff --git a/AGENTS.md b/AGENTS.md index 89fad47..a7f0e21 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -257,9 +257,9 @@ The run itself is minutes of bash and hundreds of `git` processes, and several lanes build in sibling worktrees of one checkout: the wrapper waits for any live run, takes `${TMPDIR:-/tmp}/openrepotools-pytest.lock` (`flock` where there is one, a `mkdir` lock on macOS, which has none), waits again -inside it, then runs `python3 -m pytest tests -q "$@"`. Every lane on one -workstation must name the SAME lock file or there is no lock, which is the -whole reason the path is written here as well as in the file. +inside it, then runs `python3 -m pytest tests -q "$@"` (no cache or temp left in +the checkout: #162). Every lane on one workstation must name the SAME lock file +or there is no lock, which is the whole reason the path is written here too. Two measured defects on 2026-09-14 (opensoft/openRepoTools#51), both of them inside a guard that had been copied into four briefs: diff --git a/README.md b/README.md index 3cce271..18c8f7e 100644 --- a/README.md +++ b/README.md @@ -260,7 +260,7 @@ that would not take it, and a create needs a directory's search bit as well as its write bit. A refusal creates none of those directories either. `cp` follows a symlink, and an install through one leaves the command uninstalled and writes these bytes into whatever it points at. Then a `17 of 17 placed in ` line, -and the `export PATH=…` line if that directory is not on your `PATH`. +and the `export PATH=…` line if that directory is not on your `PATH`. Last, it names this repository's virtual environment, `${XDG_CACHE_HOME:-~/.cache}/openRepoTools/venvs/openRepoTools/` (one per repository, outside the estate; #162), says whether it is there, and prints the two commands that make it: it never makes one, and `tests/run.sh` runs from it once it has `pytest`. It also places **fourteen things that are not files in that directory**: THREE SKILLS, `/handoff`, `/lane-swap` and `/restart`, each at diff --git a/docs/README-lanes.md b/docs/README-lanes.md index 69fd1b3..955d5fc 100644 --- a/docs/README-lanes.md +++ b/docs/README-lanes.md @@ -3778,6 +3778,91 @@ caught stops the run); usage is 64. goes to `${XDG_STATE_HOME:-$HOME/.local/state}/openRepoTools/lanes//scratch/` or the harness scratchpad. +### The estate report — `sweep --all --dry-run --report` + +The net under every actor that never runs `lane-end`: **creation outpacing +closeout as a number that is read every day.** It reads every lane's #97 +inventory and snapshot under `/.lane-state` (or +`$LANES_LANE_STATE_ROOT`), the workspace repository's register on +`origin/`, and the disk — **never the derived index** (Amendment 14 +clause (b)) — and CHANGES NOTHING: no fetch, no index refresh +(`GIT_OPTIONAL_LOCKS=0`), no expiry. + +```sh +lane-worktrees sweep --all --dry-run --report [--estate ] [--post | --post #] +``` + +| section | what is listed | +|---|---| +| FOREIGN repositories | a CLONE inside a worktree container (`.lane-worktrees/`, `.claude/worktrees`, `worktrees`, `-worktrees`), and a second clone of an origin the estate already has a checkout of: path, size, last commit, owner (the last `Lane:` trailer), and LOAD-BEARING with its dependents where something leans on it | +| Orphaned worktrees of ENDED lanes | trees an inventory records, and directories under `.lane-worktrees/`, of a lane whose snapshot is `CLOSED` or whose log's last lane line is `ENDED` or `RETIRED` — with the sweep that retires them | +| Unmerged branches | local branches of every checkout with no upstream, a gone upstream, a diverged one, or unpushed commits: tip, state, age, owner, and whether a worktree holds it. Never deleted; `main`, `master` and `rescue/*` are not listed | +| Root main divergence | a checkout whose local `main` is ahead of `origin/main`; where every changed path is under `handoffs/` or `lanes/`, the remedy is Amendment 4's — those live in the workspace repository and are pushed per write | +| Awaiting disposition | a `rescue/*` branch (local or on origin) and a DIRTY inventory tree whose git directory has been still for `aging_days` (14), with owner and age | +| Caches and sandboxes | `--include-caches`' and `--include-sandboxes`' rows across every lane, by size | +| Ignored directories over `ignored_report_mb` | every ignored directory of every repository at or over 50 MB (`du -sk`) | +| Evidence-shaped paths | untracked or ignored `junit*.xml`, `MANIFEST*`, `*-report.md`, `*REPORT*.md`, `canary-*`, `*-evidence`, `deployment-evidence` inside a repository, and a directory of reports beside the repositories — each a finding to move to the evidence root below | +| Sweep archives past retention | `sweep --expire`'s own dry run: what WOULD expire, and what is past retention but kept because its rescue left origin. Expiring stays an act (`sweep --expire --yes`) | +| Workspace repository hygiene | the workspace's `.gitignore` lines it lacks (with the one command that adds them), and bytecode already in its history (for a person's word: history is rewritten only on one) | +| Records the report could not read | a lane snapshot, inventory directory or record, or an inventory tree's `git status`, that could not be read — unknown is never absent, and the sections above may be missing what it holds | +| `status --all` findings | the estate command's ahead/behind, fork, shape-pin and parked-record lines, reported as what they are and never counted as dirt (`LANE_WORKTREES_STATUS` names the command when it is not beside this one) | + +The summary table at its head also carries **the count of rescue branches** and +**the sweeps directory's size**, so the cleanup's own footprint is watched. +`--post ` writes the report there; `--post /#` comments it +on that issue through `gh`; with no `--post` it is printed. `--all` and +`--report` come together, with no lane, no `--yes` and no `--expire` (usage, 64). + +**`lane-start` runs it once a day per workstation**, after the row is written +and before the launch: the first start of a UTC day takes the stamp +`${XDG_STATE_HOME:-~/.local/state}/openRepoTools/report-.stamp` +(an atomic `set -C` create, so two starts at once run one report), removes +older stamps, and starts the report DETACHED — stdin, stdout and stderr closed, +in a subshell that exits at once — over the start's own `$PROJECTS_ROOT`, +writing `…/openRepoTools/reports/.md`. Every step either works or is skipped in +silence: the report never delays a start and never fails one. `--dry-run` +starts none; `LANE_WORKTREES_REPORT=off` is the switch (both suites set it). + +### Nothing new lands beside the code + +The sweep retires what is already there; these keep more from arriving. + +* **`tests/run.sh`** sends bytecode to + `${XDG_CACHE_HOME:-~/.cache}/openRepoTools/pycache`, turns pytest's cache off, + and roots every temporary directory of a run under + `${XDG_STATE_HOME:-~/.local/state}/openRepoTools/tmp/-/`, which its + EXIT trap removes however the run ends — after stopping the suite's whole + process group, so nothing it started outlives the run root or the lock. With + a terminal on stdin the suite runs in the FOREGROUND instead, so `--pdb` or a + `breakpoint()` can read it (a background group would be stopped by SIGTTIN, + and the wrapper would wait on it holding the lock). A SIGKILL is the one end no trap sees; + the pid in the name is what lets `--include-sandboxes` tell its owner is gone. +* **One virtual environment per repository, outside the estate:** + `${XDG_CACHE_HOME:-~/.cache}/openRepoTools/venvs//`. `openRepoTools + --install` names this repository's and says whether it is there; `tests/run.sh` + runs from it when it has `pytest`. A `venv/` or `.venv/` inside a worktree is + a legacy leftover for `--include-caches`. +* **One evidence root per repository:** + `${XDG_STATE_HOME:-~/.local/state}/openRepoTools/evidence//-/` + for JUnit files, manifests, reports, canary and deployment evidence — never + inside a checkout and never beside the repositories. The report lists what is + still found inside the estate. +* **The workspace repository never carries bytecode.** `openRepoTools wip init` + seeds `__pycache__/`, `*.pyc`, `*.pyo`, `.pytest_cache/`, `.mypy_cache/`, + `.ruff_cache/`, `node_modules/`, `.venv/`, `venv/` and `site-packages/` (a + virtual environment under any other name) into the new + workspace's `.gitignore` (a line the template already carries is not + repeated). Every commit `lanes-edit.sh` makes asks git what its pathspec would + stage (`git add --dry-run`) or holds staged already, and REFUSES, exit **2**, + nothing staged, when any of it is bytecode, a cache, a dependency tree or a virtual environment — and + offers the `.gitignore` lines a workspace without them lacks. Attachments are + committed by hand, so the same question is a subcommand: + + ```sh + lanes-edit.sh pathspec-check handoffs//attachments/ + # 0 clean · 2 the offending paths on stdout, the offer on stderr · 64 usage + ``` + ## Hand edits After **any** hand edit made with an allowed tool (python read/write, `sed -i diff --git a/lane-start b/lane-start index 57ed0ee..1f2db66 100755 --- a/lane-start +++ b/lane-start @@ -4288,6 +4288,48 @@ if (( fresh )) && [ "$ri_active" = 1 ] && (( ! no_launch )) && (( ! dry_run )); esac fi +# ------------- 5e. the estate's daily report (opensoft/openRepoTools#162) +# +# ONCE A DAY PER WORKSTATION, DETACHED, AND NEVER A REASON A START WAITS OR +# FAILS. `lane-worktrees sweep --all --dry-run --report` is the net under every +# actor that never runs `lane-end`: what is FOREIGN, orphaned, unpublished, +# aging or past retention across the estate, as one table, written to the state +# directory. It changes nothing. The first start of the day takes the stamp - +# `set -C` makes taking it one atomic create, so two starts at once run one +# report - and starts it with stdin, stdout and stderr closed, in a subshell +# that exits at once, so the report is no child of the session this command is +# about to exec. Every step of this either works or is skipped in silence: +# a report that could not start is tomorrow's start's to run. +# `LANE_WORKTREES_REPORT=off` is the switch, and both suites set it. +lane_daily_report() { + [ "${LANE_WORKTREES_REPORT:-on}" = off ] && return 0 + ldr_bin="" + if [ -x "$SCRIPT_DIR/lane-worktrees" ]; then + ldr_bin="$SCRIPT_DIR/lane-worktrees" + else + ldr_bin="$(command -v lane-worktrees 2>/dev/null || :)" + fi + [ -n "$ldr_bin" ] || return 0 + ldr_state="${XDG_STATE_HOME:-$HOME/.local/state}/openRepoTools" + ldr_day="$(date -u +%Y%m%d 2>/dev/null || :)" + [ -n "$ldr_day" ] || return 0 + mkdir -p -- "$ldr_state/reports" 2>/dev/null || return 0 + ( set -C; : > "$ldr_state/report-$ldr_day.stamp" ) 2>/dev/null || return 0 + for ldr_old in "$ldr_state"/report-*.stamp; do + [ "$ldr_old" = "$ldr_state/report-$ldr_day.stamp" ] || rm -f -- "$ldr_old" 2>/dev/null || : + done + ldr_out="$ldr_state/reports/$(date -u +%Y%m%dT%H%M%SZ).md" + # THE ESTATE THIS START READS: `$PROJECTS_ROOT` as resolved above, named to + # the report rather than left for it to find again. + ldr_estate="" + [ -d "$PROJECTS_ROOT" ] && ldr_estate="$PROJECTS_ROOT" + ( "$ldr_bin" sweep --all --dry-run --report --post "$ldr_out" \ + ${ldr_estate:+--estate "$ldr_estate"} /dev/null 2>&1 & ) 2>/dev/null || : + step "the estate's daily report is running, detached: $ldr_out" + return 0 +} +(( dry_run )) || lane_daily_report || : + # ---------------------------------------------------------------- 6. launch # # `cmd` was decided in 5a, before the row was written. diff --git a/lane-worktrees b/lane-worktrees index ec1cfd7..8fd16fb 100755 --- a/lane-worktrees +++ b/lane-worktrees @@ -10,6 +10,8 @@ abandoned worktrees, branches, scratch and killed-suite sandboxes [--include-foreign --word ""] [--live ... | --live none] [--fetch] [--estate ] lane-worktrees sweep --expire [--dry-run | --yes] [--porcelain] + lane-worktrees sweep --all --dry-run --report [--post ] + [--estate ] lane-worktrees --help | --version A SWAP THAT DOES NOT FINISH GRACEFULLY LEAVES ITS WRITERS' TREES WHERE THEY @@ -48,6 +50,16 @@ origin at the rescued SHA; an archive whose rescue branch is gone or moved, or that is not whole (its MANIFEST.sha256 missing or not matching, its rescues.tsv unreadable), is never expired. +THE ESTATE REPORT (`--all --dry-run --report`) is the net under every actor +that never runs `lane-end`: FOREIGN clones, trees of ENDED lanes, unmerged +branches, root-`main` divergence, rescue branches and dirty trees awaiting +disposition, caches and sandboxes, large ignored directories, evidence-shaped +paths, archives past retention, the workspace's bytecode hygiene and +`status --all`'s findings - one markdown document, printed or `--post`ed to a +file or an issue. It changes nothing. `lane-start` runs it once a day per +workstation, detached (`LANE_WORKTREES_REPORT=off` is the switch). It exits 0, +or 1 when `--post` could not deliver it (the report is then printed). + `--yes` (and `--expire --yes`) is SWITCHED OFF until opensoft/openRepoTools #170 lands: it is refused, exit 2, unless the environment carries LANE_WORKTREES_ENABLE_YES=1 - which the test suite sets, and which is not for a @@ -2835,8 +2847,12 @@ def _gitdir_pointer(path: str, base: str, now: float, out: list) -> None: # ================================================================== expire -def expire(opts) -> int: - conf = load_conf() +def expiry_rows(conf: dict) -> list: + """(archive, age in days, keep|expire, why) for every archive under the + sweeps directory - `sweep --expire`'s table, and the report's. An archive + past retention is `expire` only while it is whole (`archive_ledger`) and + every rescue branch it records is on origin at the rescued SHA; one whose + rescue left or moved is the only copy and is kept.""" root = sweeps_dir() keep_days = conf["retention_days"] now = _dt.datetime.now(_dt.timezone.utc) @@ -2887,6 +2903,14 @@ def expire(opts) -> int: + "; ".join(missing) + " - this archive is the only copy")) else: rows.append((adir, age, "expire", f"{age} d old and every rescue branch is on origin")) + return rows + + +def expire(opts) -> int: + conf = load_conf() + root = sweeps_dir() + keep_days = conf["retention_days"] + rows = expiry_rows(conf) failed = 0 for i, (adir, age, disp, why) in enumerate(rows): if disp == "expire" and opts.yes: @@ -2973,6 +2997,559 @@ def archive_ledger(adir: str): return out, "" +# ================================================================== report + +EVIDENCE_NAME_RE = re.compile( + r"(^canary-|^deployment-evidence$|-evidence$|^evidence$|-report\.md$|REPORT[^/]*\.md$|" + r"^junit[^/]*\.xml$|^MANIFEST)", re.IGNORECASE) +HANDOFF_PATHS = ("handoffs/", "lanes/") +#: `rescue//-` names its lane; a rescue branch of another +#: convention (`rescue/`) names none. +LANE_SHAPED_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*-[0-9]+$") + + +def estate_root_default() -> str: + for cand in (os.environ.get("PROJECTS_ROOT", ""), os.path.join(_home(), "projects"), + os.path.join(_home(), "Projects")): + if cand and os.path.isdir(cand): + return cand + return "" + + +def _under_container(path: str, root: str) -> str: + """The worktree container a path sits in - `.lane-worktrees/`, + `.claude/worktrees`, `worktrees`, `-worktrees` - or ''.""" + rel = os.path.relpath(path, root) + parts = rel.split(os.sep) + for i, p in enumerate(parts[:-1]): + if p == ".lane-worktrees" and i + 1 < len(parts) - 1: + return os.sep.join(parts[:i + 2]) + if p in ("worktrees", ".worktrees") or p.endswith("-worktrees"): + return os.sep.join(parts[:i + 1]) + return "" + + +def _du(paths: list) -> dict: + """{path: bytes} through `du -sk`, which walks in C: an estate's ignored + trees are millions of files, and a Python walk of them is minutes.""" + out: dict = {} + for i in range(0, len(paths), 64): + chunk = paths[i:i + 64] + proc = run(["du", "-sk", "--", *chunk], timeout=600) + for line in proc.stdout.splitlines(): + size, _, path = line.partition("\t") + if size.isdigit(): + out[path] = int(size) * 1024 + return out + + +def _lane_parents(root: str, repos: list) -> list: + """Every directory a lane's control root and worktree root can sit in. + #97 puts `.lane-state/` - and lane-start `.lane-worktrees/` - + beside the lane's recorded `dir`, so a lane whose checkout is nested + (`/InkRouter/IRRS`) keeps them in `/InkRouter`, not at + the estate's top.""" + out = [root] if root else [] + for r in repos: + parent = os.path.dirname(r.rstrip(os.sep)) + if parent and parent not in out: + out.append(parent) + return out + + +def _lane_state_dirs(parents: list) -> list: + dirs = [] + cands = [os.environ.get("LANES_LANE_STATE_ROOT", "")] + cands += [os.path.join(p, ".lane-state") for p in parents] + for cand in cands: + if cand and os.path.isdir(cand) and cand not in dirs: + dirs.append(cand) + return dirs + + +def _lane_tree_roots(parents: list) -> list: + """`/.lane-worktrees`, each that exists.""" + out = [] + for p in parents: + cand = os.path.join(p, ".lane-worktrees") + if os.path.isdir(cand) and cand not in out: + out.append(cand) + return out + + +class Report: + """`sweep --all --dry-run --report`: the estate's leftovers as one + document. It reads every lane's #97 inventory, the workspace register and + the disk; it changes nothing.""" + + def __init__(self, opts): + self.opts = opts + self.conf = load_conf() + self.root = opts.estate or estate_root_default() + self.helper = Helper() + self.sections: list = [] + self.counts: dict = {} + self.now = time.time() + self.unreadable: dict = {} + + def add(self, title: str, rows: list, empty: str, note_text: str = "") -> None: + self.sections.append((title, rows, empty, note_text)) + self.counts[title] = len(rows) + + # ------------------------------------------------------------- reading + def build(self) -> None: + repos = estate_repos(self.root) if self.root else [] + self.deps = dependents_map(repos) + clones = [r for r in repos if os.path.isdir(os.path.join(r, ".git"))] + canonical = self._canonical(clones) + self.canonical = canonical + parents = _lane_parents(self.root, canonical) + self.state_dirs = _lane_state_dirs(parents) + self.tree_roots = _lane_tree_roots(parents) + self._foreign(clones, canonical) + closed = self._closed_lanes() + self._orphans(closed) + self._branches(canonical) + self._root_main(canonical) + self._aging(canonical) + self._caches_and_sandboxes() + self._ignored_and_evidence(repos, canonical) + self._archives() + self._workspace_hygiene() + self._status_all() + self.add("Records the report could not read", [[p, w] for p, w in + sorted(self.unreadable.items())], "none", + "UNKNOWN IS NEVER ABSENT: a lane snapshot, inventory or tree the report " + "could not read is one whose findings above may be missing.") + + # A READ THAT FAILED IS A FINDING, never an empty answer (Amendment 7(d)): + # an inventory the report could not read would otherwise make it clean. + def _ls(self, d: str) -> list: + try: + return sorted(os.listdir(d)) + except (FileNotFoundError, NotADirectoryError): + return [] + except OSError as exc: + self.unreadable.setdefault(d, f"could not be listed ({exc.strerror or exc})") + return [] + + def _record(self, path: str, need_path: bool = False) -> dict: + try: + with open(path, encoding="utf-8") as fh: + text = fh.read() + except FileNotFoundError: + return {} + except (OSError, UnicodeDecodeError) as exc: + self.unreadable.setdefault(path, "could not be read (" + + (getattr(exc, "strerror", "") or str(exc)) + ")") + return {} + out = {} + for line in text.splitlines(): + key, sep, value = line.partition(": ") + if sep: + out[key.strip()] = value.strip() + if need_path and not out.get("path"): + self.unreadable.setdefault(path, "an inventory record that names no path") + return out + + def _tree_records(self, tdir: str) -> list: + return [self._record(os.path.join(tdir, f), need_path=True) for f in self._ls(tdir) + if f.endswith(".yaml") and not f.startswith(".")] + + def _canonical(self, clones: list) -> list: + """The estate's own checkouts: a clone outside every worktree + container. Of several clones of one origin, the one whose folder is + named for the repository is the checkout and the rest are duplicates.""" + by_origin: dict = {} + for c in clones: + if _under_container(c, self.root): + continue + url = git_out(c, "config", "--get", "remote.origin.url") or "" + key = (slug_of(url) or url).lower().rstrip("/") + by_origin.setdefault(key, []).append(c) + self.duplicates: dict = {} + out = [] + for key, group in by_origin.items(): + if not key or len(group) == 1: + out.extend(group) + continue + name = key.rsplit("/", 1)[-1].replace(".git", "") + named = [g for g in group if os.path.basename(g).lower() == name] + # ONE CHECKOUT PER ORIGIN, CHOSEN DETERMINISTICALLY: the one named + # for the repository, the shallowest of several so named, and the + # shallowest of all where none is. The rest are duplicates - never + # all of them canonical, which would hide the finding. + pool = named or group + keep = sorted(pool, key=lambda g: (g.count(os.sep), g))[0] + out.append(keep) + for g in group: + if g != keep: + self.duplicates[g] = (keep, len(named) != 1) + return sorted(out) + + def _foreign(self, clones: list, canonical: list) -> None: + rows = [] + for c in clones: + container = _under_container(c, self.root) + why = "" + if container: + why = f"a CLONE inside {container}, where a lane makes worktrees" + elif c in self.duplicates: + first, ambiguous = self.duplicates[c] + why = f"a second clone of {first}'s origin" + ( + " (no single clone is named for the repository; the shallowest is " + "taken as its checkout)" if ambiguous else "") + if not why: + continue + last = git_out(c, "log", "-1", "--format=%cs") or "?" + msg = git_out(c, "log", "-1", "--format=%B") or "" + owner = (re.findall(r"^Lane:\s*(\S+)", msg, re.MULTILINE) or ["-"])[-1] + lean = leaned_on(c, git_common_dir(c), self.deps) + rows.append([c, why, human_size(dir_size(c)), last, owner, + ("LOAD-BEARING: " + _lean_text(lean)) if lean else ""]) + self.add("FOREIGN repositories", rows, "none: every clone is the estate's own checkout", + "A lane creates worktrees, never clones. Each is a person's to dispose of.") + + def _closed_lanes(self) -> dict: + """{lane: why} for every lane whose snapshot is CLOSED or whose log's + last lane-kind line is ENDED or RETIRED.""" + closed: dict = {} + for d in self.state_dirs: + for lane in self._ls(d): + if self._record(os.path.join(d, lane, "lane-state.yaml")).get("state") == "CLOSED": + closed[lane] = "its snapshot is CLOSED" + wroot = self.helper.workspace_root() if self.helper.path else "" + if wroot: + branch = os.environ.get("LANES_BRANCH", "main") + proc = git(wroot, "grep", "-E", "-e", + "^(STARTED|RESUMED|PAUSED|ENDED|RETIRED) — lane ", + f"origin/{branch}", "--", "lanes/log") + last: dict = {} + for line in proc.stdout.splitlines(): + _ref, _, rest = line.partition(":") + fname, _, text = rest.partition(":") + last[os.path.basename(fname)[:-3]] = text.split(" ", 1)[0] + for lane, verb in last.items(): + if verb in ("ENDED", "RETIRED"): + closed.setdefault(lane, f"its log ends {verb}") + return closed + + def _orphans(self, closed: dict) -> None: + rows = [] + seen = set() + for d in self.state_dirs: + for lane in sorted(closed): + for rec in self._tree_records(os.path.join(d, lane, "trees")): + path = rec.get("path", "") + if path and os.path.isdir(path) and real(path) not in seen: + seen.add(real(path)) + rows.append([path, lane, closed[lane], "lane-worktrees sweep " + lane]) + for lane, lroot in [(ln, r) for ln in sorted(closed) for r in self.tree_roots]: + ldir = os.path.join(lroot, lane) + for n in self._ls(ldir): + full = os.path.join(ldir, n) + if os.path.isdir(full) and real(full) not in seen: + seen.add(real(full)) + rows.append([full, lane, closed[lane], f"lane-worktrees sweep {lane} " + "--include-scratch --include-foreign --word \"\""]) + self.add("Orphaned worktrees of ENDED lanes", rows, "none") + + def _branches(self, canonical: list) -> None: + rows = [] + fmt = ("%(refname:short)%1f%(objectname:short=12)%1f%(upstream:short)%1f" + "%(upstream:track)%1f%(committerdate:iso-strict)%1f%(contents:trailers)%00") + for c in canonical: + proc = git(c, "for-each-ref", f"--format={fmt}", "refs/heads/") + if proc.returncode != 0: + continue + held = {r.get("branch", "")[len("refs/heads/"):] + for r in worktree_registrations(c) if r.get("branch")} + for rec in proc.stdout.split("\0"): + cols = rec.strip("\n").split("\x1f") + if len(cols) < 6: + continue + name, tip, up, track, date, trailers = cols[:6] + if name in PROTECTED_BRANCHES or name.startswith("rescue/"): + continue + if not up: + state = "no upstream" + elif "gone" in track: + state = f"upstream {up} gone" + elif "ahead" in track and "behind" in track: + state = f"diverged from {up} {track}" + elif "ahead" in track: + state = f"unpushed {track}" + else: + continue + owner = (re.findall(r"^Lane:\s*(\S+)", trailers, re.MULTILINE) or ["-"])[-1] + rows.append([f"{c} {name}", tip, state, f"{_age_days(date)} d", + owner, "held by a worktree" if name in held else ""]) + rows.sort(key=lambda r: -int(r[3].split()[0])) + self.add("Unmerged branches with a missing, diverged or unpushed upstream", rows, + "none", "Listed, never deleted: a person pushes each to rescue//... " + "or gives a word.") + + def _root_main(self, canonical: list) -> None: + rows = [] + for c in canonical: + if git(c, "rev-parse", "-q", "--verify", "refs/heads/main").returncode != 0 or \ + git(c, "rev-parse", "-q", "--verify", "refs/remotes/origin/main").returncode != 0: + continue + n = git_out(c, "rev-list", "--count", "refs/remotes/origin/main..refs/heads/main") or "0" + if not n.isdigit() or int(n) == 0: + continue + paths = (git_out(c, "diff", "--name-only", "refs/remotes/origin/main...refs/heads/main") + or "").splitlines() + only = bool(paths) and all(p.startswith(HANDOFF_PATHS) for p in paths) + rows.append([c, f"local main ahead of origin by {n}", + "only handoff/register paths" if only else f"{len(paths)} path(s)", + "Amendment 4: handoffs and the register live in the workspace " + "repository and are pushed per write" if only else + "push it through a pull request, or a person's word"]) + self.add("Root main divergence", rows, "none") + + def _aging(self, canonical: list) -> None: + aging = self.conf["aging_days"] + rows = [] + rescue_count = 0 + seen = set() + for c in canonical: + proc = git(c, "for-each-ref", "--format=%(refname)\t%(committerdate:iso-strict)", + "refs/heads/rescue/", "refs/remotes/origin/rescue/") + for line in proc.stdout.splitlines(): + ref, _, date = line.partition("\t") + name = ref.split("/rescue/", 1)[1] + if (c, name) in seen: + continue + seen.add((c, name)) + rescue_count += 1 + age = _age_days(date) + if age >= aging: + first = name.split("/", 1)[0] if "/" in name else "" + owner = first if LANE_SHAPED_RE.match(first) else "-" + rows.append([f"{c} rescue/{name}", f"rescue branch, {age} d", + owner, "awaiting disposition"]) + for d in self.state_dirs: + for lane in self._ls(d): + for rec in self._tree_records(os.path.join(d, lane, "trees")): + path = rec.get("path", "") + if not path or not os.path.isdir(path): + continue + dirt = git_out(path, "status", "--porcelain") + if dirt is None: + self.unreadable.setdefault(path, f"lane {lane}'s inventory tree; git " + "status could not be read in it") + continue + if not dirt: + continue + newest = _gitdir_activity(path) + age = int((self.now - newest) // 86400) if newest else 0 + if age >= aging: + rows.append([path, f"dirty inventory tree, {age} d", lane, + "awaiting disposition"]) + self.rescue_count = rescue_count + self.add(f"Awaiting disposition (older than {aging} days)", rows, "none") + + def _caches_and_sandboxes(self) -> None: + rows = [] + roots = [] + for lroot in getattr(self, "tree_roots", []): + try: + roots += [os.path.join(lroot, n) for n in sorted(os.listdir(lroot))] + except OSError: + pass + for c in getattr(self, "canonical", []): + wt = os.path.join(c, ".claude", "worktrees") + if os.path.isdir(wt): + roots.append(wt) + for root in roots: + for path, size in find_caches(root): + rows.append([path, "cache", size]) + for it in sandbox_items(ProcScan(), self.conf): + rows.append([it.path, "sandbox", it.size]) + rows.sort(key=lambda r: -r[2]) + total = sum(r[2] for r in rows) + self.add(f"Caches and sandboxes ({human_size(total)} in all)", + [[p, k, human_size(s)] for p, k, s in rows], "none", + "Removed without archiving by `--include-caches` / `--include-sandboxes`.") + + def _ignored_and_evidence(self, repos: list, canonical: list) -> None: + threshold = self.conf["ignored_report_mb"] * 1024 * 1024 + big, evidence = [], [] + for r in repos: + proc = git(r, "ls-files", "--others", "--ignored", "--exclude-standard", + "--directory", "-z") + ignored = [p.rstrip("/") for p in proc.stdout.split("\0") if p.endswith("/")] + sizes = _du([os.path.join(r, p) for p in ignored]) if ignored else {} + for path, size in sizes.items(): + if size >= threshold: + big.append([path, human_size(size)]) + unt = git(r, "ls-files", "--others", "--exclude-standard", "--directory", "-z") + for p in (unt.stdout.split("\0") + [i + "/" for i in ignored]): + name = p.rstrip("/").rsplit("/", 1)[-1] + if p and EVIDENCE_NAME_RE.search(name): + evidence.append([os.path.join(r, p.rstrip("/")), "untracked or ignored in " + + os.path.basename(r)]) + if self.root: + repo_set = set(repos) + for n in sorted(os.listdir(self.root)): + full = os.path.join(self.root, n) + if full in repo_set or not os.path.isdir(full) or n.startswith(".lane-") \ + or n.endswith("-worktrees"): + continue + if EVIDENCE_NAME_RE.search(n) or self._holds_reports(full): + evidence.append([full, "a directory of reports beside the repositories"]) + big.sort(key=lambda r: r[0]) + self.add(f"Ignored directories over {self.conf['ignored_report_mb']} MB", big, "none") + self.add("Evidence-shaped paths inside the estate", evidence, "none", + "One evidence root per repository: ${XDG_STATE_HOME:-$HOME/.local/state}/" + "openRepoTools/evidence//-/. Each is a finding to move.") + + @staticmethod + def _holds_reports(d: str) -> bool: + try: + for e in os.scandir(d): + if e.is_file() and EVIDENCE_NAME_RE.search(e.name): + return True + if e.is_dir(follow_symlinks=False) and not e.name.startswith("."): + for f in os.scandir(e.path): + if f.is_file() and EVIDENCE_NAME_RE.search(f.name): + return True + except OSError: + return False + return False + + def _archives(self) -> None: + """`sweep --expire`'s dry run, folded in: what WOULD expire, and what + is past retention but kept because its rescue left origin. Expiring + stays an act (`sweep --expire --yes`); the report changes nothing.""" + root = sweeps_dir() + keep = self.conf["retention_days"] + rows = [] + for adir, age, disp, why in expiry_rows(self.conf): + if disp == "expire": + rows.append([adir, f"{age} d", "would expire: lane-worktrees sweep --expire --yes"]) + elif age >= keep: + rows.append([adir, f"{age} d", "kept past retention: " + why]) + self.sweeps_size = dir_size(root) if os.path.isdir(root) else 0 + self.add(f"Sweep archives past retention ({keep} days)", rows, "none") + + def _workspace_hygiene(self) -> None: + rows = [] + wroot = self.helper.workspace_root() if self.helper.path else "" + if wroot: + try: + with open(os.path.join(wroot, ".gitignore"), encoding="utf-8") as fh: + have = set(fh.read().splitlines()) + except OSError: + have = set() + want = ["__pycache__/", "*.pyc", "*.pyo", ".pytest_cache/", ".mypy_cache/", + ".ruff_cache/", "node_modules/", ".venv/", "venv/", "site-packages/"] + missing = [w for w in want if w not in have] + if missing: + rows.append([os.path.join(wroot, ".gitignore"), "lacks " + " ".join(missing), + "printf '%s\\n' " + " ".join(f"'{m}'" for m in missing) + + f" >> {wroot}/.gitignore"]) + hist = git(wroot, "log", "--all", "--format=", "--name-only", "--", + "*.pyc", "*__pycache__*") + paths = sorted({p for p in hist.stdout.splitlines() if p}) + if paths: + rows.append([wroot, f"{len(paths)} bytecode path(s) in history, e.g. {paths[0]}", + "a person's word: history is rewritten only on one"]) + self.add("Workspace repository hygiene", rows, "none") + + def _status_all(self) -> None: + # `LANE_WORKTREES_STATUS` names the estate command when it is not + # beside this one (a test names a fake; a person, their own build). + exe = os.environ.get("LANE_WORKTREES_STATUS", "") + if not exe: + exe = os.path.join(os.path.dirname(os.path.realpath(__file__)), "status") + if not os.access(exe, os.X_OK): + exe = shutil.which("status") or "" + rows = [] + why = "" + if not exe: + why = "no `status` beside this command or on PATH" + else: + env = dict(os.environ) + if self.root: + # THE ESTATE THIS REPORT READ, however it was resolved: from + # `--estate`, `$PROJECTS_ROOT` or the home default. + env["PROJECTS_DIR"] = self.root + proc = run([exe, "--all"], env=env, timeout=600) + if proc.returncode not in (0, 1): + why = f"`status --all` exited {proc.returncode}: " + _first_line(proc.stderr) + context = "" + for line in proc.stdout.splitlines(): + s = line.strip() + if line.startswith("=== "): + context = s.strip("= ").replace("status ", "") + elif s.startswith("- "): + rows.append([context, s[2:]]) + self.add("`status --all` findings (pin drift and tracking are reported as what they " + "are, never counted as dirt)", rows, "none", why) + + # ------------------------------------------------------------ writing + def markdown(self) -> str: + host = os.environ.get("LANES_WORKSTATION") or socket.gethostname() + out = [f"# Estate report - {host} - {utc_now()}", "", + f"`{PROG} sweep --all --dry-run --report` over `{self.root or '(no estate)'}`. " + "It changed nothing.", "", + "| finding | count |", "|---|---|"] + for title, rows, _e, _n in self.sections: + out.append(f"| {title} | {len(rows)} |") + out.append(f"| rescue branches (all ages) | {getattr(self, 'rescue_count', 0)} |") + out.append(f"| the sweeps directory | {human_size(getattr(self, 'sweeps_size', 0))} |") + for title, rows, empty, note_text in self.sections: + out += ["", f"## {title}", ""] + if note_text: + out += [note_text, ""] + if not rows: + out.append(f"_{empty}_") + continue + for r in rows[:500]: + out.append("- " + " · ".join(str(c).replace("\n", " ") for c in r if c != "")) + if len(rows) > 500: + out.append(f"- ... and {len(rows) - 500} more") + return "\n".join(out) + "\n" + + +def report_all(opts) -> int: + rep = Report(opts) + rep.build() + text = rep.markdown() + post = opts.post or "" + if not post: + sys.stdout.write(text) + return 0 + m = re.match(r"^([A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+)#(\d+)$", post) + if m: + fd, tmp = tempfile.mkstemp(prefix="lane-worktrees-report.", suffix=".md") + with os.fdopen(fd, "w", encoding="utf-8") as fh: + fh.write(text) + try: + proc = run(["gh", "issue", "comment", m.group(2), "--repo", m.group(1), + "--body-file", tmp], timeout=120) + finally: + os.unlink(tmp) + if proc.returncode != 0: + note(f"the report could not be posted to {post}: " + _first_line(proc.stderr)) + sys.stdout.write(text) + return 1 + say(f"{PROG}: the report was posted to {post}: {proc.stdout.strip()}") + return 0 + try: + os.makedirs(os.path.dirname(os.path.abspath(post)), exist_ok=True) + with open(post, "w", encoding="utf-8") as fh: + fh.write(text) + except OSError as exc: + note(f"the report could not be written to {post}: {exc}") + sys.stdout.write(text) + return 1 + say(f"{PROG}: the report is at {post}") + return 0 + + # ==================================================================== main class _Parser(argparse.ArgumentParser): @@ -3000,6 +3577,9 @@ def parse(argv: list): parser.add_argument("--fetch", action="store_true") parser.add_argument("--estate") parser.add_argument("--expire", action="store_true") + parser.add_argument("--all", action="store_true") + parser.add_argument("--report", action="store_true") + parser.add_argument("--post") parser.add_argument("-h", "--help", action="store_true") parser.add_argument("--version", action="store_true") return parser.parse_args(argv) @@ -3056,6 +3636,12 @@ def _main(argv: list) -> int: if ns.verb != "sweep": note("usage: lane-worktrees sweep [--dry-run | --yes] ... (see --help)") return 64 + if ns.all or ns.report or ns.post is not None: + if not (ns.all and ns.report) or ns.lane or ns.yes or ns.expire: + note("usage: the estate-wide sweep only reports: lane-worktrees sweep --all " + "--dry-run --report [--post ]") + return 64 + return report_all(ns) # `--yes` IS SWITCHED OFF until the data-loss paths the reviews found are # closed (opensoft/openRepoTools#170). The dry run, the porcelain #163's # gate reads and the report stay whole; acting takes the switch, which the diff --git a/lanes-edit.sh b/lanes-edit.sh index 34ed9a6..7588406 100755 --- a/lanes-edit.sh +++ b/lanes-edit.sh @@ -362,6 +362,7 @@ # lanes-edit.sh lane-tree-now # lanes-edit.sh lane-reconcile # lanes-edit.sh worktrees [ | --all] [--index] [--fetch] +# lanes-edit.sh pathspec-check ... # opensoft/openRepoTools#162 # # A lane is RUNNING, SWAPPING, SWAPPED or CLOSED, and WHICH OF THOSE IT IS # WITH NO LIVE HOLDER is what says where its session stopped: `RUNNING` with @@ -428,7 +429,9 @@ # checkout that cannot be rebased, — Amendment 15 — a register holding # two rows whose lane names differ only by case, which every writer and # `canon-lane` refuse until 15(d)'s merge, or AMENDMENT 12's BLOCKED -# PROMPT. SIX MEANINGS ON ONE NUMBER, and they +# PROMPT, or — opensoft/openRepoTools#162 — a commit whose pathspec +# would stage bytecode, a cache or a virtual environment +# (`commit_push`, `pathspec-check`). SEVEN MEANINGS ON ONE NUMBER, and they # stay readable only because of where each can occur: bad arguments to a # subcommand that predates Amendment 11; `register-row`'s "not lane-shaped"; # AN UNKNOWN SUBCOMMAND (the `*)` arm below), which is how a caller detects @@ -1856,6 +1859,71 @@ git_timeout_die() { # die "timed out after ${GIT_TIMEOUT}s on '$1' — the mutex is released and your commit is local; nothing was lost." 3 } +# BYTECODE NEVER RIDES A WORKSPACE COMMIT (opensoft/openRepoTools#162, cause +# 19: a handoff's attachments were committed with a whole virtualenv's +# `__pycache__` inside them, and the workspace repository carries them for +# ever). What `git add` WOULD stage for a pathspec is asked of git itself +# (`add --dry-run`, which honours `.gitignore` exactly as the real add does), +# and every path in that answer with a cache, bytecode or environment +# component is printed. Empty means the pathspec is clean. +# ONE PER LINE AND READ WITH `read`, never word-split: `*.pyc` unquoted is a +# glob, and the shell would expand it against whatever directory this runs in. +BYTECODE_IGNORES='__pycache__/ +*.pyc +*.pyo +.pytest_cache/ +.mypy_cache/ +.ruff_cache/ +node_modules/ +.venv/ +venv/ +site-packages/' +bytecode_in_pathspec() { # ... + bip_repo="${1-}"; shift || : + [ "$#" -gt 0 ] || return 0 + # WHAT THE ADD WOULD STAGE, AND WHAT IS STAGED ALREADY: `add --dry-run` says + # nothing of a path the index already holds at these bytes, and the commit + # takes it all the same. A staged DELETION of bytecode is a cleanup and is + # let through (`--diff-filter=d`). + { + git -C "$bip_repo" add --dry-run --ignore-missing -- "$@" 2>/dev/null | + sed -n "s/^add '\(.*\)'\$/\1/p" + git -C "$bip_repo" diff --cached --name-only --diff-filter=d -- "$@" 2>/dev/null + } | + awk '!seen[$0]++ { + n = split($0, part, "/"); hit = 0 + for (i = 1; i <= n; i++) + if (part[i] == "__pycache__" || part[i] == ".pytest_cache" || + part[i] == ".mypy_cache" || part[i] == ".ruff_cache" || + part[i] == "node_modules" || part[i] == ".venv" || + part[i] == "venv" || part[i] == "site-packages") hit = 1 + if ($0 ~ /\.py[co]$/) hit = 1 + if (hit) print }' +} + +# The refusal's words, and the OFFER beside them: the ignore lines the +# workspace's own `.gitignore` lacks, with the one command that adds them. +# `openRepoTools wip init` seeds them into a new workspace; an older one is +# offered them here, where they are missing. +bytecode_refusal_text() { # + brt_n="$(printf '%s\n' "${1-}" | awk 'NF { n++ } END { print n + 0 }')" + brt_missing="" + while IFS= read -r brt_l; do + [ -n "$brt_l" ] || continue + command grep -Fqx -- "$brt_l" "$LANES_REPO/.gitignore" 2>/dev/null || + brt_missing="$brt_missing '$brt_l'" + done <> %s/.gitignore\n' \ + "$brt_missing" "$brt_missing" "$LANES_REPO" + fi + printf 'Nothing was staged or committed. Unstage or delete those paths, then re-run.' +} + # commit_push "" [...] — the pathspecs default to the # register alone, which is every caller that predates Amendment 7. A LANDING # or LANDED writes TWO (the register and the lane's log) so that both halves @@ -1923,6 +1991,8 @@ commit_push() { || die "git update-index --force-remove $cp_ren failed — the log is already renamed on disk and git still holds the old path, so nothing was committed. Re-run; if it refuses again, \`git -C $LANES_REPO rm --cached -- $cp_ren\` is the same act by hand." 6 fi if [ "$cp_add_n" -gt 0 ]; then + cp_byte="$(bytecode_in_pathspec "$LANES_REPO" ${cp_add[@]+"${cp_add[@]}"})" + [ -z "$cp_byte" ] || die "$(bytecode_refusal_text "$cp_byte")" 2 git -C "$LANES_REPO" -c "$CP_EXACT" add -- ${cp_add[@]+"${cp_add[@]}"} || die "git add failed" 6 fi if git -C "$LANES_REPO" -c "$CP_EXACT" diff --cached --quiet -- "${CP_PATHS[@]}"; then @@ -16923,7 +16993,24 @@ EOF fi ;; + # opensoft/openRepoTools#162 — WHAT A HAND COMMIT OF A HANDOFF'S ATTACHMENTS + # WOULD STAGE THAT IS BYTECODE, asked BEFORE the commit. `commit_push` + # refuses such a pathspec for every write this file makes; attachments are + # committed by hand, so this is the same question for a person to ask first. + # 0 nothing in those paths would stage bytecode, a cache or an environment + # 2 something would: the paths on stdout, the offer on stderr + # 64 usage + pathspec-check) + [ "$#" -gt 0 ] || die "usage: pathspec-check ... (relative to the workspace repository $LANES_REPO)" 64 + pck_out="$(bytecode_in_pathspec "$LANES_REPO" "$@")" + if [ -n "$pck_out" ]; then + printf '%s\n' "$pck_out" + note "$(bytecode_refusal_text "$pck_out")" + exit 2 + fi + ;; + *) - die "unknown subcommand '$cmd' (verify-row|set-row-state|append-row-status|replace-in-row|append-session-id|append-line|add-row|retire-rows|archive-rows|migrate-state-cells|commit|rename-lane|log|claim|release|who|history|swapped|session-start|guard|idle-holders|live-holder|window-session|transcript-holders|binding|request-handoff|session-lane|window-lane|lane-dir|lane-profile|lane-agent|lane-transcript|lane-last|workspace-root|last-session|forks|duplicate-holder|workstation|fetch-age|lanes|lane-groups|next-free|sibling-filter|resolve-repo|lane-objects|register-row|register-row-local|retired-identity|canon-lane|resolve-home|managed-projection|lane-state|set-lane-state|lane-trees|set-lane-tree|lane-tree-now|lane-reconcile|lane-holders|legacy-restart-check|publish-handoff|restart-intent|set-restart-intent|worktrees)" 2 + die "unknown subcommand '$cmd' (verify-row|set-row-state|append-row-status|replace-in-row|append-session-id|append-line|add-row|retire-rows|archive-rows|migrate-state-cells|commit|rename-lane|log|claim|release|who|history|swapped|session-start|guard|idle-holders|live-holder|window-session|transcript-holders|binding|request-handoff|session-lane|window-lane|lane-dir|lane-profile|lane-agent|lane-transcript|lane-last|workspace-root|last-session|forks|duplicate-holder|workstation|fetch-age|lanes|lane-groups|next-free|sibling-filter|resolve-repo|lane-objects|register-row|register-row-local|retired-identity|canon-lane|resolve-home|managed-projection|lane-state|set-lane-state|lane-trees|set-lane-tree|lane-tree-now|lane-reconcile|lane-holders|legacy-restart-check|publish-handoff|restart-intent|set-restart-intent|worktrees|pathspec-check)" 2 ;; esac diff --git a/openRepoTools b/openRepoTools index 0294954..1af8831 100755 --- a/openRepoTools +++ b/openRepoTools @@ -1204,6 +1204,26 @@ install_commands() { say " export PATH=\"$dir:\$PATH\"" ;; esac + venv_note +} + +# ONE VIRTUAL ENVIRONMENT PER REPOSITORY, OUTSIDE THE ESTATE +# (opensoft/openRepoTools#162, cause 3: "968 of 1,000 removals were dependency +# caches"). `${XDG_CACHE_HOME:-$HOME/.cache}/openRepoTools/venvs//` is +# the convention `docs/README-lanes.md` states, and this repository's own is +# NAMED here with its state - and NOT CREATED: making one needs pip and the +# network, and an install from a checkout needs neither. `tests/run.sh` runs +# the suite from it once it is there and has pytest. +venv_note() { + local venv="${XDG_CACHE_HOME:-$HOME/.cache}/openRepoTools/venvs/openRepoTools" + if [ -x "$venv/bin/python3" ]; then + say "openRepoTools: this repository's virtual environment is $venv (present; tests/run.sh runs from it when it has pytest)" + else + say "openRepoTools: this repository's virtual environment is $venv (not created; one per repository, outside the estate):" + # QUOTED FOR THE SHELL IT IS PASTED INTO: `$XDG_CACHE_HOME` is the + # person's to choose, spaces and all. + say " python3 -m venv $(printf '%q' "$venv") && $(printf '%q' "$venv/bin/python3") -m pip install pytest" + fi } @@ -2425,9 +2445,50 @@ wip_substitute() { never a blank, and never a literal {{…}} written into your repository. Nothing was created." fi + # THE WORKSPACE'S `.gitignore` CARRIES THE BYTECODE AND CACHE RULES + # (opensoft/openRepoTools#162, cause 19: generated bytecode committed into + # handoff attachments). Added HERE, in the one function both step 7 paths + # go through, so the bytes a seed writes and the bytes a re-run compares + # an earlier run's leftover against are the same bytes. A line the template + # already carries is not repeated. + case "${file##*/}" in + tpl-.gitignore) wip_hygiene_lines "$file.subst" >>"$file.subst" || + die "could not add the bytecode and cache rules to the template file $file" ;; + esac mv -f -- "$file.subst" "$file" } +#: The lines every `-wip` ignores (#162): bytecode, test and type-check +#: caches, dependency trees and virtual environments - a handoff attachment +#: is evidence, and none of these is. +WIP_HYGIENE_LINES=( + "__pycache__/" + "*.pyc" + "*.pyo" + ".pytest_cache/" + ".mypy_cache/" + ".ruff_cache/" + "node_modules/" + ".venv/" + "venv/" + "site-packages/" +) + +# The hygiene block, less every line `$1` already carries; nothing at all when +# it carries them all. +wip_hygiene_lines() { # + local line missing="" + for line in "${WIP_HYGIENE_LINES[@]}"; do + command grep -Fqx -- "$line" "$1" 2>/dev/null || missing="$missing$line +" + done + [ -n "$missing" ] || return 0 + printf '\n# Bytecode, caches, dependency trees and virtual environments are never\n' + printf '# a workspace repository'"'"'s: a handoff attachment is evidence, not a build\n' + printf '# (opensoft/openRepoTools#162). `lanes-edit.sh` refuses to commit them.\n' + printf '%s' "$missing" +} + # The template's bytes, materialized EXACTLY the way `--install` gets its own # files: from the checkout beside this command when run from one, else # `gh api …/contents/` at the pinned ref, else the raw URL. *That the diff --git a/tests/conftest.py b/tests/conftest.py index e2c9292..0f53a61 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -36,6 +36,11 @@ #: builds every environment it runs from scratch and decides for itself. os.environ["LANES_INDEX"] = "off" +#: opensoft/openRepoTools#162 — and no `lane-start` a test runs starts the +#: estate's daily report: it is detached, it reads the whole estate, and the +#: state directory it stamps may be the workstation's own. +os.environ["LANE_WORKTREES_REPORT"] = "off" + #: The pinned openRepoShape checkout. Every path this suite reads out of the #: standard hangs off this one name, so a bump of the pin moves one line in #: `contracts/openreposhape-pin.yaml` and the gitlink beside it, and nothing diff --git a/tests/run.sh b/tests/run.sh index f3f1835..29023cc 100755 --- a/tests/run.sh +++ b/tests/run.sh @@ -42,6 +42,29 @@ # atomicity POSIX guarantees; the holder's pid is left inside it so a lock a # killed run left behind is taken over rather than waited on for ever. # +# NOTHING NEW LANDS BESIDE THE CODE (opensoft/openRepoTools#162, causes 3 and +# 4). A suite run used to leave `tests/__pycache__`, `.pytest_cache` and a +# killed run's sandboxes behind in whichever worktree ran it, and a lane's +# leftovers were 968 of 1,000 removals in the estate's cleanup. So: +# +# * BYTECODE goes to `${XDG_CACHE_HOME:-$HOME/.cache}/openRepoTools/pycache` +# (`PYTHONPYCACHEPREFIX`), never into a checkout; +# * pytest's cache is off (`-p no:cacheprovider`); +# * EVERY TEMPORARY DIRECTORY of the run - pytest's `--basetemp` and the +# `TMPDIR` the suite's own `mktemp`s use - is under ONE run root, +# `${XDG_STATE_HOME:-$HOME/.local/state}/openRepoTools/tmp/-/`, +# which an EXIT trap removes however the run ends. A SIGKILL is the one +# end no trap sees; the pid in the root's name is what lets +# `lane-worktrees sweep --include-sandboxes` tell its owner is gone. +# * the suite runs from the repository's own virtual environment where one +# is there and has pytest (`${XDG_CACHE_HOME:-$HOME/.cache}/openRepoTools/ +# venvs/openRepoTools`, which `openRepoTools --install` names), put FIRST +# on PATH so the command line still reads `python3 -m pytest` - the shape +# every other lane's poll counts. +# +# THE LOCK IS COMPUTED BEFORE `TMPDIR` IS MOVED: it is the workstation's, and +# a run that locked a file under its own run root would lock nothing. +# # bash 3.2 (Apple's stock `/bin/bash`) parses this file in CI, like every other # bash file here: no `${x,,}`, no `mapfile`, no `declare -A`, no `local -n`. @@ -79,13 +102,106 @@ wait_for_pytest() { # — block while any other suite is running return 0 } +# THE RUN ROOT AND ITS CLEANUP. `cleanup` is the EXIT trap: it stops whatever +# is left of the suite's PROCESS GROUP, then removes the run root and, on the +# `mkdir` path, the lock it holds. The INT and TERM handlers stop the suite and +# EXIT, so the EXIT trap runs - a handler that only cleaned up and returned +# would release the lock while the suite carried on. +# +# THE GROUP, NOT THE PID: pytest is waiting on a shell suite, a `git`, a +# fixture's `sleep` when the signal comes, and a TERM to pytest alone leaves +# them running with their temp root deleted under them and the lock released +# around them. The suite is started in a process group of its own (below), and +# the whole group is stopped and waited for, KILLed after ten seconds. +RUN_ROOT="" +LOCKDIR_HELD="" +SUITE_PID="" +SUITE_PGID="" +# A member still RUNNING, read from `ps` and never from `kill -0`: a zombie +# answers `kill -0` until whoever adopted it reaps it, which may be never. +group_alive() { + ps -A -o pgid= -o stat= 2>/dev/null | + awk -v g="$SUITE_PGID" '$1 == g && $2 !~ /^Z/ { n++ } END { exit n ? 0 : 1 }' +} +stop_group() { + [ -n "$SUITE_PGID" ] || return 0 + kill -TERM -- "-$SUITE_PGID" 2>/dev/null || return 0 + if [ -n "$SUITE_PID" ]; then wait "$SUITE_PID" 2>/dev/null || :; fi + sg_n=0 + while group_alive && [ "$sg_n" -lt 50 ]; do + sleep 0.2 + sg_n=$((sg_n + 1)) + done + kill -KILL -- "-$SUITE_PGID" 2>/dev/null || : + return 0 +} +cleanup() { + stop_group + [ -n "$RUN_ROOT" ] && rm -rf -- "$RUN_ROOT" + [ -n "$LOCKDIR_HELD" ] && rm -rf -- "$LOCKDIR_HELD" + return 0 +} +on_signal() { # + stop_group + exit "$1" +} + +make_run_root() { + run_base="${XDG_STATE_HOME:-$HOME/.local/state}/openRepoTools/tmp" + RUN_ROOT="$run_base/$(date -u +%Y%m%dT%H%M%SZ)-$$" + mkdir -p -- "$RUN_ROOT/tmp" "$RUN_ROOT/basetemp" || { + printf '%s: cannot create the run root %s\n' "$prog" "$RUN_ROOT" >&2 + exit 1 + } + chmod 700 "$RUN_ROOT" 2>/dev/null || : + export TMPDIR="$RUN_ROOT/tmp" + export PYTHONPYCACHEPREFIX="${XDG_CACHE_HOME:-$HOME/.cache}/openRepoTools/pycache" + venv="${XDG_CACHE_HOME:-$HOME/.cache}/openRepoTools/venvs/openRepoTools" + if [ -x "$venv/bin/python3" ] && "$venv/bin/python3" -c 'import pytest' >/dev/null 2>&1; then + PATH="$venv/bin:$PATH"; export PATH + printf '%s: the suite runs from the repository venv %s\n' "$prog" "$venv" >&2 + fi +} + # `${1+"$@"}` AND NEVER A BARE `"$@"`: under `set -u`, BASH 3.2 — Apple's stock # shell, and the one this file's `mkdir` lock exists for — treats `"$@"` with no # positional parameters as an unbound variable and exits. `"${*:-}"` is the same # rule for the line that echoes them. +# +# THE SUITE RUNS IN THE BACKGROUND AND IS WAITED FOR, because bash runs a trap +# only once its FOREGROUND child returns: a TERM sent to this wrapper while +# pytest ran in the foreground waited out the whole suite. `wait` is +# interrupted by a trapped signal; the handler then stops the suite itself. +# `<&0` keeps the suite's stdin this shell's: a background command's default +# stdin, with job control off, is /dev/null. +# +# `set -m` FOR THE ONE LINE THAT STARTS IT puts the suite in a process group of +# its own, whose id is its pid, so `stop_group` reaches every process it +# started (one that made a session or group of its own is beyond any wrapper). +# Monitor mode is off again before the `wait`, so no job notice is printed. +# +# A TERMINAL ON STDIN RUNS THE SUITE IN THE FOREGROUND instead, in this +# wrapper's own process group - the terminal's foreground group. A background +# group reading the terminal (`--pdb`, `breakpoint()`, `input()`) is stopped by +# SIGTTIN, and this wrapper would wait on it for ever holding the +# workstation's lock. In the foreground a Ctrl-C reaches every process of the +# suite from the terminal itself; the cost is that a TERM sent to this wrapper +# from elsewhere is acted on once the suite returns. run_suite() { - printf '%s: python3 -m pytest tests -q %s\n' "$prog" "${*:-}" >&2 - python3 -m pytest tests -q ${1+"$@"} + printf '%s: python3 -m pytest tests -q -p no:cacheprovider %s\n' "$prog" "${*:-}" >&2 + if [ -t 0 ]; then + python3 -m pytest tests -q -p no:cacheprovider --basetemp="$RUN_ROOT/basetemp" ${1+"$@"} + return $? + fi + set -m + python3 -m pytest tests -q -p no:cacheprovider --basetemp="$RUN_ROOT/basetemp" ${1+"$@"} <&0 & + SUITE_PID=$! + SUITE_PGID=$SUITE_PID + set +m + wait "$SUITE_PID" + rs_rc=$? + SUITE_PID="" + return "$rs_rc" } wait_for_pytest "before the lock" @@ -95,7 +211,11 @@ if command -v flock >/dev/null 2>&1; then # by its exit — including a kill, which no `rm` in a trap would survive. exec 9> "$LOCK" || { printf '%s: cannot open the lock file %s\n' "$prog" "$LOCK" >&2; exit 1; } flock 9 || { printf '%s: could not take the lock %s\n' "$prog" "$LOCK" >&2; exit 1; } + trap cleanup EXIT + trap 'on_signal 130' INT + trap 'on_signal 143' TERM wait_for_pytest "inside the lock" + make_run_root run_suite ${1+"$@"} exit $? fi @@ -162,9 +282,11 @@ printf '%s\n' "$$" > "$LOCKDIR/pid" 2>/dev/null || : # THE SIGNAL HANDLERS EXIT, and that is the whole point of writing them out # separately: a handler that only cleans up and RETURNS releases the lock while # this suite carries on running, which is the collision the lock exists for. -trap 'rm -rf -- "$LOCKDIR"' EXIT -trap 'rm -rf -- "$LOCKDIR"; exit 130' INT -trap 'rm -rf -- "$LOCKDIR"; exit 143' TERM +LOCKDIR_HELD="$LOCKDIR" +trap cleanup EXIT +trap 'on_signal 130' INT +trap 'on_signal 143' TERM wait_for_pytest "inside the lock" +make_run_root run_suite ${1+"$@"} exit $? diff --git a/tests/test_lane_helpers.sh b/tests/test_lane_helpers.sh index c4e81a7..93c3c75 100755 --- a/tests/test_lane_helpers.sh +++ b/tests/test_lane_helpers.sh @@ -467,6 +467,12 @@ export LANES_NO_GITHUB=1 # index turns it back on for its own cases. export LANES_INDEX=off +# opensoft/openRepoTools#162 — AND NO CASE STARTS THE ESTATE REPORT EITHER. +# `lane-start` runs `lane-worktrees sweep --all --report` detached, once a day, +# and the state directory it stamps may be the workstation's own. Off for the +# file; the section about the report turns it on for its own cases. +export LANE_WORKTREES_REPORT=off + # Amendment 8, ruling (g): the harness exports `CLAUDE_CODE_SESSION_ID` into # every shell a session runs, INCLUDING the one running this suite. It is a # holder test now, so a case that does not set it deliberately must not inherit @@ -15565,6 +15571,61 @@ run "$E" restart-intent repoSV-3 hasnt "…the lane was never marked ready on a launch nothing proved" "$out" "$(printf 'state\tready')" has "…the deadline is in the record for a reader in another window" "$out" "INDETERMINATE" +# ========================================================================= +echo "== opensoft/openRepoTools#162: bytecode never rides a workspace commit ==" +# +# A handoff's attachments were committed with a whole virtualenv's +# `__pycache__` inside them (cause 19). `pathspec-check` is the question a +# hand commit asks first, and `commit_push` asks it of every write this file +# makes: what `git add` WOULD stage that is bytecode, a cache or an +# environment. The `.gitignore` lines are OFFERED where the workspace lacks +# them, and once they are there the same pathspec stages nothing of the kind. +B162_REL="handoffs/sandbox/attachments/b162" +B162="$WIP/$B162_REL" +mkdir -p "$B162/r1/__pycache__" "$B162/r1/venv/lib/python3.12/site-packages" \ + "$B162/r1/env-x/lib/python3.12/site-packages" +printf 'findings\n' > "$B162/r1/notes.md" +printf 'bytecode\n' > "$B162/r1/__pycache__/m.cpython-312.pyc" +printf 'pkg\n' > "$B162/r1/venv/lib/python3.12/site-packages/pkg.py" +printf 'pkg\n' > "$B162/r1/env-x/lib/python3.12/site-packages/other.py" +B162_IGN="" +[ -f "$WIP/.gitignore" ] && B162_IGN="$(cat "$WIP/.gitignore")" +run "$E" pathspec-check "$B162_REL" +is "pathspec-check refuses an attachment that carries bytecode" "$rc" 2 +has "…naming the bytecode" "$out" "$B162_REL/r1/__pycache__/m.cpython-312.pyc" +has "…and the virtual environment" "$out" "$B162_REL/r1/venv/lib/python3.12/site-packages/pkg.py" +hasnt "…and not the attachment's own notes" "$out" "notes.md" +has "…and offers the ignore lines the workspace lacks" "$err" "'__pycache__/'" +has "…a venv under any name included, by its site-packages" "$out" "$B162_REL/r1/env-x/lib/python3.12/site-packages/other.py" +has "…and the line that covers it is among those offered" "$err" "'site-packages/'" +run "$E" pathspec-check "$B162_REL/r1/notes.md" +is "a pathspec with nothing of the kind passes" "$rc" 0 +b162_head="$(git -C "$WIP" rev-parse HEAD)" +run env LANES_LANE=repoA-1 LANES_PATH="$B162_REL" "$E" commit "attach r1" +is "the commit path refuses the same pathspec" "$rc" 2 +has "…saying what it is" "$err" "none of them is ever a workspace repository's" +is "…and commits nothing" "$(git -C "$WIP" rev-parse HEAD)" "$b162_head" +is "…and stages nothing" "$(git -C "$WIP" diff --cached --name-only | grep -c . || :)" 0 +printf '%s\n' '__pycache__/' '*.pyc' 'venv/' >> "$WIP/.gitignore" +run "$E" pathspec-check "$B162_REL" +is "without the site-packages line a venv under another name is still refused" "$rc" 2 +printf '%s\n' 'site-packages/' >> "$WIP/.gitignore" +run "$E" pathspec-check "$B162_REL" +is "with the offered lines in place it stages nothing of the kind" "$rc" 0 +# ALREADY STAGED (Copilot round 2 on #169): `add --dry-run` is silent about a +# path the index already holds, and the commit would carry it regardless. +git -C "$WIP" add -f -- "$B162_REL/r1/__pycache__/m.cpython-312.pyc" +run "$E" pathspec-check "$B162_REL" +is "bytecode already staged is refused too" "$rc" 2 +has "…and named" "$out" "$B162_REL/r1/__pycache__/m.cpython-312.pyc" +b162_head="$(git -C "$WIP" rev-parse HEAD)" +run env LANES_LANE=repoA-1 LANES_PATH="$B162_REL" "$E" commit "attach r1" +is "…by the commit path as well" "$rc" 2 +is "…which commits nothing" "$(git -C "$WIP" rev-parse HEAD)" "$b162_head" +git -C "$WIP" reset -q -- "$B162_REL" +rm -rf "$WIP/handoffs/sandbox" +if [ -n "$B162_IGN" ]; then printf '%s\n' "$B162_IGN" > "$WIP/.gitignore"; else rm -f "$WIP/.gitignore"; fi +git -C "$WIP" checkout -q -- .gitignore 2>/dev/null || : # ------------------------------------------------------- nothing real touched # diff --git a/tests/test_lane_worktrees.py b/tests/test_lane_worktrees.py index cedbbb0..28ca0a9 100644 --- a/tests/test_lane_worktrees.py +++ b/tests/test_lane_worktrees.py @@ -1172,8 +1172,20 @@ def test_a_path_that_is_not_utf8_is_a_row_not_a_traceback(estate): """B2: a worktree named in Latin-1 is reported - exit inside 0/3/2.""" estate.lane_root.mkdir(parents=True, exist_ok=True) bad = os.fsencode(str(estate.lane_root)) + b"/caf\xe9" - subprocess.run(["git", "-C", str(estate.checkout), "worktree", "add", "-q", "-b", "feat/latin1", - bad, "origin/main"], env=estate.env, check=True, capture_output=True) + # A FILESYSTEM THAT REFUSES THE NAME HAS NO SUCH WORKTREE TO REPORT (#177). + # APFS, the macOS runner's, answers a non-UTF-8 name with EILSEQ ("Illegal + # byte sequence"), and git's own `.git/worktrees/caf\xe9` meets the same + # refusal. The directory is made first so the refusal is the filesystem's + # and not a fixture traceback; where it is made, the case stays live. + try: + os.mkdir(bad) + except OSError as exc: + pytest.skip(f"filesystem refuses non-UTF-8 names: {exc}") + add = subprocess.run(["git", "-C", str(estate.checkout), "worktree", "add", "-q", "-b", + "feat/latin1", bad, "origin/main"], env=estate.env, capture_output=True) + if add.returncode != 0: + pytest.skip("filesystem refuses non-UTF-8 names: git worktree add exited " + f"{add.returncode}: {add.stderr.decode('utf-8', 'replace').strip()}") estate.write_spec() for args in (("--dry-run", "--porcelain"), ("--dry-run",)): proc = subprocess.run([sys.executable, str(LW), "sweep", LANE, *args], capture_output=True, diff --git a/tests/test_lane_worktrees_report.py b/tests/test_lane_worktrees_report.py new file mode 100644 index 0000000..60d80db --- /dev/null +++ b/tests/test_lane_worktrees_report.py @@ -0,0 +1,416 @@ +# SPDX-License-Identifier: Apache-2.0 +"""`lane-worktrees sweep --all --dry-run --report` and the start that runs it +once a day (opensoft/openRepoTools#162, the "nightly report" fold). + +The report is the net under every actor that never runs `lane-end`. Held here: + + * every section the issue names finds what it is for, in an estate built + under `tmp_path`: a FOREIGN clone (inside a worktree container, and a + second clone of one origin), an orphaned tree of an ENDED lane, unmerged + branches with a missing or diverged upstream, root-`main` divergence that + is handoff and register paths only, a rescue branch and a dirty inventory + tree past the aging threshold, caches and a killed suite's sandbox, + ignored directories over the threshold, evidence-shaped paths, archives + past retention, the workspace's `.gitignore` and its bytecode history, + and `status --all`'s findings; + * it changes nothing - the estate is snapshotted around it; + * `--post ` writes it, `--post owner/repo#n` comments it through `gh`; + * `lane-start` runs it detached, once a day per workstation, and a report + that is slow, failing or switched off neither delays nor fails a start. + +The `lane-start` cases reuse `tests/test_lane_start_claude_current.py`'s +sandbox: a real `lane-start` against a register in `tmp_path`. +""" + +from __future__ import annotations + +import datetime as _dt +import os +import shutil +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +from conftest import REPO, WINDOWS_SKIP +import test_lane_worktrees as LW +import test_lane_start_claude_current as LS + +pytestmark = [WINDOWS_SKIP, pytest.mark.skipif( + shutil.which("git") is None or shutil.which("bash") is None, + reason="the report reads git, and lane-start is bash")] + +FAKE_STATUS = """#!/bin/sh +cat <<'OUT' +status --all: reading every estate: + +=== status repoA === + root main /x/repoA + - shape pin 71cf5de is 1 commit(s) behind opensoft/openRepoShape main +OUT +printf ' - estate read at %s\n' "${PROJECTS_DIR:-unset}" +exit 1 +""" + + +def _commit_files(e, cwd, message, when, files): + """A commit whose author and committer dates are `when`.""" + for rel, text in files.items(): + path = Path(cwd) / rel + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text) + e.git("add", "-A", cwd=cwd) + e.git("commit", "-q", "-m", message, cwd=cwd, + env={"GIT_COMMITTER_DATE": when, "GIT_AUTHOR_DATE": when}) + return e.git("rev-parse", "HEAD", cwd=cwd) + + +def build_estate(e) -> dict: + """An estate with one finding of every kind the report lists.""" + f = {} + # FOREIGN: a clone inside the lane's worktree root, and a second clone of + # repoA's origin beside the repositories. + f["clone"] = e.lane_root / "cloned" + e.git("clone", "-q", LW.GH_URL, f["clone"], cwd=e.root) + f["dup"] = e.projects / "repoA-copy" + e.git("clone", "-q", LW.GH_URL, f["dup"], cwd=e.root) + # ... and two clones of one origin NEITHER of which is named for it: one + # is still the checkout, the other still a finding. + c_origin = e.root / "remotes" / "repoC.git" + e.git("init", "-q", "--bare", "-b", "main", c_origin, cwd=e.root) + for name in ("c-one", "c-two"): + e.git("clone", "-q", c_origin, e.projects / name, cwd=e.root) + f["c-two"] = e.projects / "c-two" + # ORPHANED: a tree in the inventory of a lane whose snapshot is CLOSED. + f["orphan"] = e.projects / ".lane-worktrees" / "repoA-9" / "left" + e.git("worktree", "add", "-q", "-b", "feat/left", f["orphan"], "origin/main") + closed = e.projects / ".lane-state" / "repoA-9" + (closed / "trees").mkdir(parents=True) + (closed / "lane-state.yaml").write_text("schema: 1\nstate: CLOSED\n") + (closed / "trees" / "c1.yaml").write_text(f"schema: 1\npath: {f['orphan']}\n") + # ... and one of a lane whose checkout is NESTED: #97 keeps its control + # root, and lane-start its worktree root, beside the checkout's own + # parent, `projects/group/`, never at the estate's top. + nested_origin = e.root / "remotes" / "repoB.git" + e.git("init", "-q", "--bare", "-b", "main", nested_origin, cwd=e.root) + e.git("clone", "-q", nested_origin, e.projects / "group" / "repoB", cwd=e.root) + group_closed = e.projects / "group" / ".lane-state" / "repoB-2" + group_closed.mkdir(parents=True) + (group_closed / "lane-state.yaml").write_text("schema: 1\nstate: CLOSED\n") + f["nested-orphan"] = e.projects / "group" / ".lane-worktrees" / "repoB-2" / "leftover" + f["nested-orphan"].mkdir(parents=True) + # UNMERGED: no upstream at all, and an upstream that diverged. + e.git("branch", "--no-track", "feat/nowhere", "origin/main") + e.git("checkout", "-q", "-b", "feat/diverged", "origin/main") + e.commit(e.checkout, "d1", {"d.txt": "1\n"}) + e.git("push", "-q", "-u", "origin", "feat/diverged") + e.git("reset", "-q", "--hard", "HEAD~1") + e.commit(e.checkout, "d2", {"d.txt": "2\n"}) + e.git("checkout", "-q", "main") + # ROOT MAIN DIVERGENCE: local main ahead by a handoff only. + e.commit(e.checkout, "handoff(repoA-1@Eagle): x", {"handoffs/repoA/h.md": "h\n"}) + # AGING: a rescue branch 30 days old, and a dirty inventory tree of a live + # lane whose git directory has been still for 30 days. + e.git("checkout", "-q", "-b", "rescue/repoA-1/old-20260101T000000Z", "origin/main") + _commit_files(e, e.checkout, "old rescue", (_dt.datetime.now(_dt.timezone.utc) + - _dt.timedelta(days=30)).strftime("%Y-%m-%dT%H:%M:%SZ"), {"r.txt": "r\n"}) + e.git("checkout", "-q", "main") + f["stale-dirty"] = e.lane_root / "stale" + e.git("worktree", "add", "-q", "-b", "feat/stale", f["stale-dirty"], "origin/main") + (f["stale-dirty"] / "wip.txt").write_text("unsaved\n") + live = e.projects / ".lane-state" / LW.LANE / "trees" + live.mkdir(parents=True) + (live / "s.yaml").write_text(f"schema: 1\npath: {f['stale-dirty']}\n") + old = time.time() - 30 * 86400 + gitdir = Path(e.git("rev-parse", "--absolute-git-dir", cwd=f["stale-dirty"])) + for name in ("index", "HEAD", "logs/HEAD"): + if (gitdir / name).exists(): + os.utime(gitdir / name, (old, old)) + # CACHES and a killed suite's SANDBOX. + (e.lane_root / "x-scratch" / "__pycache__").mkdir(parents=True) + (e.lane_root / "x-scratch" / "__pycache__" / "a.pyc").write_bytes(b"\0" * 2048) + f["sandbox"] = e.sandboxes / "tmp.killedrun1" + f["sandbox"].mkdir() + (f["sandbox"] / "f").write_text("x") + os.utime(f["sandbox"], (old, old)) + # IGNORED over the threshold (0 MB in this estate's sweep.conf), and + # EVIDENCE: an untracked JUnit file, and a directory of reports. + (e.checkout / "build").mkdir() + (e.checkout / "build" / "out.bin").write_bytes(b"\0" * 4096) + (e.checkout / ".gitignore").write_text((e.checkout / ".gitignore").read_text() + "build/\n") + (e.checkout / "junit-results.xml").write_text("\n") + f["reports"] = e.projects / "cleanup-20261005" + f["reports"].mkdir() + (f["reports"] / "FINAL-REPORT.md").write_text("# report\n") + # PAST RETENTION: an archive 120 days old that would expire, and one 150 + # days old kept because the rescue branch it records left origin. + f["archive"] = LW._archive(e, 120, None) + f["kept"] = LW._archive(e, 150, "rescue/repoA-1/gone-1") + # THE WORKSPACE: a .gitignore without the bytecode lines, and a .pyc in + # its history. + wip_origin = e.root / "wip.git" + wip = e.projects / "wip" + e.git("init", "-q", "--bare", "-b", "main", wip_origin, cwd=e.root) + e.git("clone", "-q", wip_origin, wip, cwd=e.root) + (wip / "lanes" / "log").mkdir(parents=True) + (wip / "lanes" / "LANES.md").write_text("# register\n") + (wip / "lanes" / "log" / "repoA-9.md").write_text( + f"STARTED — lane repoA-9, session {LW.ME}@Eagle, 2026-09-01T00:00:00Z, lane:repoA-9\n" + f"ENDED — lane repoA-9, session {LW.ME}@Eagle, 2026-09-02T00:00:00Z, lane:repoA-9\n") + (wip / ".gitignore").write_text("*.swp\n") + (wip / "handoffs" / "a" / "__pycache__").mkdir(parents=True) + (wip / "handoffs" / "a" / "__pycache__" / "m.cpython-312.pyc").write_bytes(b"\0") + e.git("add", "-A", cwd=wip) + e.git("commit", "-q", "-m", "seed", cwd=wip) + e.git("push", "-q", "origin", "main", cwd=wip) + e.workspace = str(wip) + f["wip"] = wip + (e.fakebin / "status").write_text(FAKE_STATUS) + (e.fakebin / "status").chmod(0o755) + e.env["LANE_WORKTREES_STATUS"] = str(e.fakebin / "status") + conf = e.root / "sweep.conf" + conf.write_text("ignored_report_mb=0\n") + e.env["LANE_WORKTREES_CONF"] = str(conf) + return f + + +def report(e, *args, env=None, estate=True): + e.write_spec() + child = dict(e.env) + child.update(env or {}) + where = ["--estate", str(e.projects)] if estate else [] + return subprocess.run([sys.executable, str(LW.LW), "sweep", "--all", "--dry-run", "--report", + *where, *args], + capture_output=True, text=True, env=child, timeout=300, + cwd=str(e.root), stdin=subprocess.DEVNULL) + + +def section(text: str, title: str) -> str: + start = text.index(f"## {title}") + end = text.find("\n## ", start + 3) + return text[start:end if end != -1 else len(text)] + + +def test_the_report_finds_every_kind_of_leftover_and_changes_nothing(tmp_path): + e = LW.Estate(tmp_path) + f = build_estate(e) + before = LW.snapshot(e.root, skip=("helper.log", "helper-spec.json", "prs.json")) + proc = report(e) + after = LW.snapshot(e.root, skip=("helper.log", "helper-spec.json", "prs.json")) + assert proc.returncode == 0, proc.stdout + proc.stderr + out = proc.stdout + assert out.startswith("# Estate report") + foreign = section(out, "FOREIGN repositories") + assert str(f["clone"]) in foreign and "a CLONE inside" in foreign + assert str(f["dup"]) in foreign and "a second clone of" in foreign + assert f"{e.checkout} ·" not in foreign, "the estate's own checkout is not foreign" + assert f"{f['c-two']} · a second clone of {e.projects / 'c-one'}'s origin (no single" \ + in foreign + assert f"{e.projects / 'c-one'} ·" not in foreign + orphans = section(out, "Orphaned worktrees of ENDED lanes") + assert str(f["orphan"]) in orphans and "repoA-9" in orphans + assert f"{f['nested-orphan']} \u00b7 repoB-2 \u00b7 its snapshot is CLOSED" in orphans + branches = section(out, "Unmerged branches") + assert "feat/nowhere" in branches and "no upstream" in branches + assert "feat/diverged" in branches and "diverged from origin/feat/diverged" in branches + assert "rescue/repoA-1/old" not in branches + root_main = section(out, "Root main divergence") + assert "ahead of origin by 1" in root_main and "only handoff/register paths" in root_main + aging = section(out, "Awaiting disposition") + assert "rescue/repoA-1/old-20260101T000000Z \u00b7 rescue branch, 30 d \u00b7 repoA-1" in aging + assert str(f["stale-dirty"]) in aging and "dirty inventory tree" in aging + caches = section(out, "Caches and sandboxes") + assert "x-scratch/__pycache__" in caches and str(f["sandbox"]) in caches + ignored = section(out, "Ignored directories over") + assert str(e.checkout / "build") in ignored + evidence = section(out, "Evidence-shaped paths") + assert "junit-results.xml" in evidence and str(f["reports"]) in evidence + archives = section(out, "Sweep archives past retention") + assert f"{f['archive']} \u00b7 120 d \u00b7 would expire" in archives + assert f"{f['kept']} \u00b7 150 d \u00b7 kept past retention" in archives + assert "gone from origin" in archives + hygiene = section(out, "Workspace repository hygiene") + assert "lacks __pycache__/" in hygiene and "bytecode path(s) in history" in hygiene + status = section(out, "`status --all` findings") + assert "shape pin 71cf5de is 1 commit(s) behind" in status + assert f"estate read at {e.projects}" in status + assert "| rescue branches (all ages) | 1 |" in out + assert before == after, sorted(set(before.items()) ^ set(after.items()))[:10] + + +def test_status_reads_the_estate_the_report_resolved(tmp_path): + """Copilot on #169: with no `--estate` - lane-start's daily run - the + report resolves `$PROJECTS_ROOT`, and `status --all` is pointed at that + same estate, never at its own default.""" + e = LW.Estate(tmp_path) + build_estate(e) + proc = report(e, estate=False) + assert proc.returncode == 0, proc.stderr + assert f"over `{e.projects}`" in proc.stdout + assert f"estate read at {e.projects}" in section(proc.stdout, "`status --all` findings") + + +def test_a_record_the_report_cannot_read_is_a_finding_never_absence(tmp_path): + """Copilot round 2 on #169: an inventory or snapshot the report could not + read would otherwise make it clean.""" + e = LW.Estate(tmp_path) + build_estate(e) + trees = e.projects / ".lane-state" / "repoA-5" / "trees" + trees.mkdir(parents=True) + (trees / "c1.yaml").write_text("schema: 1\n") + locked = e.projects / ".lane-state" / "repoA-6" + locked.mkdir() + (locked / "lane-state.yaml").write_text("schema: 1\nstate: CLOSED\n") + (locked / "lane-state.yaml").chmod(0) + try: + proc = report(e) + finally: + (locked / "lane-state.yaml").chmod(0o644) + assert proc.returncode == 0, proc.stderr + found = section(proc.stdout, "Records the report could not read") + assert f"{trees / 'c1.yaml'} \u00b7 an inventory record that names no path" in found + if os.geteuid() != 0: + assert f"{locked / 'lane-state.yaml'} \u00b7 could not be read" in found + assert "| Records the report could not read | " in proc.stdout + + +def test_post_writes_a_file_or_comments_on_an_issue(tmp_path): + e = LW.Estate(tmp_path) + build_estate(e) + target = e.state / "openRepoTools" / "reports" / "today.md" + proc = report(e, "--post", str(target)) + assert proc.returncode == 0, proc.stderr + assert target.read_text().startswith("# Estate report") + log = e.root / "gh.log" + (e.fakebin / "gh").write_text( + "#!/bin/sh\nprintf '%s\\n' \"$*\" >> \"$FAKE_GH_LOG\"\n" + "[ \"$1 $2\" = 'issue comment' ] && { cat \"$7\" >> \"$FAKE_GH_LOG\"; " + "echo https://github.com/opensoft/repoA/issues/9#c1; exit 0; }\nexit 1\n") + proc = report(e, "--post", "opensoft/repoA#9", env={"FAKE_GH_LOG": str(log)}) + assert proc.returncode == 0, proc.stderr + text = log.read_text() + assert "issue comment 9 --repo opensoft/repoA --body-file" in text + assert "# Estate report" in text + + +def test_the_report_is_only_a_report(tmp_path): + e = LW.Estate(tmp_path) + for argv in (["--all", "--report", "--yes"], ["--all"], ["--report"], + [LW.LANE, "--all", "--report"]): + e.write_spec() + proc = subprocess.run([sys.executable, str(LW.LW), "sweep", *argv], capture_output=True, + text=True, env=e.env, timeout=60) + assert proc.returncode == 64, (argv, proc.stdout, proc.stderr) + + +# ================================================== lane-start runs it daily + +FAKE_REPORTER = """#!/bin/sh +printf '%s\\n' "$*" >> "$FAKE_REPORT_LOG" +if [ -n "${FAKE_REPORT_SLEEP:-}" ]; then + sleep "$FAKE_REPORT_SLEEP" & + echo "$!" > "$FAKE_REPORT_LOG.sleeper" + wait + echo finished >> "$FAKE_REPORT_LOG" +fi +exit "${FAKE_REPORT_RC:-0}" +""" + + +def _reporter(box, state: Path) -> Path: + log = box.root / "report.log" + LS._write(box.bin / "lane-worktrees", FAKE_REPORTER) + box.env.update(LANE_WORKTREES_REPORT="on", XDG_STATE_HOME=str(state), + FAKE_REPORT_LOG=str(log)) + return log + + +def _no_report_state(state: Path) -> bool: + """No stamp and no reports directory: the report's own state only - the + start writes other state of its own there (#164's index nudge).""" + base = state / "openRepoTools" + return not list(base.glob("report-*.stamp")) and not (base / "reports").exists() + + +def _wait_for(path: Path, timeout: float = 15) -> str: + deadline = time.time() + timeout + while time.time() < deadline: + if path.exists() and path.read_text(): + return path.read_text() + time.sleep(0.1) + return "" + + +def test_lane_start_runs_the_report_detached_once_a_day(tmp_path): + box = LS.Sandbox(tmp_path) + state = tmp_path / "state" + log = _reporter(box, state) + try: + result = box.start("--no-launch", FAKE_REPORT_SLEEP="90") + assert result.returncode == 0, result.stderr + calls = _wait_for(log) + # The report sleeps 90 s and then writes `finished`: a start that + # waited on it would have returned after that line. + assert "finished" not in calls, "the start waited on the report" + finally: + sleeper = Path(str(log) + ".sleeper") + if _wait_for(sleeper, 5).strip().isdigit(): + try: + os.kill(int(sleeper.read_text()), 15) + except OSError: + pass + assert calls.startswith("sweep --all --dry-run --report --post "), calls + assert f"{state}/openRepoTools/reports/" in calls + assert calls.rstrip().endswith(f"--estate {box.home / 'projects'}"), calls + day = _dt.datetime.now(_dt.timezone.utc).strftime("%Y%m%d") + assert (state / "openRepoTools" / f"report-{day}.stamp").exists() + assert "the estate's daily report is running, detached" in result.stderr + + +def test_a_stamp_from_today_runs_no_second_report_and_yesterdays_is_replaced(tmp_path): + box = LS.Sandbox(tmp_path) + state = tmp_path / "state" + log = _reporter(box, state) + day = _dt.datetime.now(_dt.timezone.utc).strftime("%Y%m%d") + (state / "openRepoTools").mkdir(parents=True) + (state / "openRepoTools" / f"report-{day}.stamp").write_text("") + result = box.start("--no-launch") + assert result.returncode == 0, result.stderr + time.sleep(1) + assert not log.exists() or log.read_text() == "", "a second report ran today" + (state / "openRepoTools" / f"report-{day}.stamp").unlink() + (state / "openRepoTools" / "report-20200101.stamp").write_text("") + box2 = LS.Sandbox(tmp_path / "second") + log2 = _reporter(box2, state) + assert box2.start("--no-launch").returncode == 0 + assert _wait_for(log2) + assert not (state / "openRepoTools" / "report-20200101.stamp").exists() + + +def test_a_failing_or_switched_off_report_never_fails_a_start(tmp_path): + box = LS.Sandbox(tmp_path) + state = tmp_path / "state" + log = _reporter(box, state) + result = box.start("--no-launch", FAKE_REPORT_RC="1") + assert result.returncode == 0, result.stderr + assert _wait_for(log), "the failing reporter was never run" + off = LS.Sandbox(tmp_path / "off") + off_log = _reporter(off, tmp_path / "off-state") + result = off.start("--no-launch", LANE_WORKTREES_REPORT="off") + assert result.returncode == 0, result.stderr + time.sleep(1) + assert not off_log.exists() + assert _no_report_state(tmp_path / "off-state") + + +def test_a_dry_run_start_runs_no_report(tmp_path): + box = LS.Sandbox(tmp_path) + state = tmp_path / "state" + log = _reporter(box, state) + result = box.start("--dry-run") + assert result.returncode == 0, result.stderr + time.sleep(1) + assert not log.exists() + assert _no_report_state(state) diff --git a/tests/test_openrepotools_command.py b/tests/test_openrepotools_command.py index 8e4d9ed..fe8f147 100644 --- a/tests/test_openrepotools_command.py +++ b/tests/test_openrepotools_command.py @@ -27,6 +27,7 @@ import hashlib import os import re +import shlex import shutil import signal import stat @@ -2374,3 +2375,38 @@ def test_a_fetching_install_follows_the_ref_it_was_given(offline_github, assert result.returncode == 0, result.stderr + result.stdout for name in INSTALLED: assert (bin_dir / name).is_file(), name + + +def test_install_names_the_repository_venv_and_whether_it_is_there(tmp_path): + """One virtual environment per repository, outside the estate (#162): + `--install` names this repository's, says whether it is there, and - when + it is not - prints the two commands that make it. It never makes one: that + needs pip and the network, and an install from a checkout needs neither.""" + cache = tmp_path / "cache" + venv = cache / "openRepoTools" / "venvs" / "openRepoTools" + first = run_cmd("--install", home=tmp_path, env={"XDG_CACHE_HOME": str(cache)}) + assert first.returncode == 0, first.stderr + assert (f"openRepoTools: this repository's virtual environment is {venv} " + "(not created; one per repository, outside the estate):") in first.stdout + assert f"python3 -m venv {venv} && {venv}/bin/python3 -m pip install pytest" in first.stdout + assert not venv.exists(), "the install made a venv" + (venv / "bin").mkdir(parents=True) + (venv / "bin" / "python3").write_text("#!/bin/sh\nexit 0\n") + (venv / "bin" / "python3").chmod(0o755) + second = run_cmd("--install", home=tmp_path, env={"XDG_CACHE_HOME": str(cache)}) + assert second.returncode == 0, second.stderr + assert f"virtual environment is {venv} (present;" in second.stdout + + +def test_the_printed_venv_command_survives_a_path_with_a_space(tmp_path): + """Copilot round 2 on #169: the command is pasted into a shell, and the + cache directory is the person's to choose.""" + cache = tmp_path / "my cache" + venv = cache / "openRepoTools" / "venvs" / "openRepoTools" + result = run_cmd("--install", home=tmp_path, env={"XDG_CACHE_HOME": str(cache)}) + assert result.returncode == 0, result.stderr + (line,) = [ln.strip() for ln in result.stdout.splitlines() + if ln.strip().startswith("python3 -m venv ")] + words = shlex.split(line) + assert words[:4] == ["python3", "-m", "venv", str(venv)], words + assert words[5] == str(venv / "bin" / "python3"), words diff --git a/tests/test_run_wrapper.py b/tests/test_run_wrapper.py new file mode 100644 index 0000000..a4ed952 --- /dev/null +++ b/tests/test_run_wrapper.py @@ -0,0 +1,284 @@ +# SPDX-License-Identifier: Apache-2.0 +"""`tests/run.sh` leaves nothing beside the code (opensoft/openRepoTools#162). + +The wrapper's lock is `test_repo_hygiene.py`'s to pin; what is held here is +what #162 added around it: + + * bytecode goes to `${XDG_CACHE_HOME:-$HOME/.cache}/openRepoTools/pycache` + and pytest's cache is off, so no `__pycache__` or `.pytest_cache` lands in + a worktree; + * every temporary directory of a run is under ONE run root, + `${XDG_STATE_HOME:-$HOME/.local/state}/openRepoTools/tmp/-/`, + and the run root is gone when the run ends - however it ends: a pass, a + failure, a TERM or an INT in the middle of the suite; + * the lock stays where every lane names it, computed BEFORE `TMPDIR` moves; + * the `mkdir` lock (macOS has no `flock`) is released the same way; + * the repository's virtual environment is used when it has pytest, with the + command line still reading `python3 -m pytest`. + +THE SUITE ITSELF IS A STUB. A fake `python3` records how it was called and +sleeps or exits on cue, and a fake `pgrep` reports no other run - the real one +would see THIS suite's own `python3 -m pytest` and wait for ever. +""" + +from __future__ import annotations + +import os +import shutil +import signal +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +from conftest import REPO, WINDOWS_SKIP + +pytestmark = [WINDOWS_SKIP, pytest.mark.skipif(shutil.which("bash") is None, + reason="the wrapper is bash")] + +WRAPPER = REPO / "tests" / "run.sh" + +FAKE_SUITE = """#!/bin/sh +# Records its own argv, the two environment values the wrapper sets, its pid +# and where it was found, then sleeps or exits on cue. +{ + printf 'argv0 %s\\n' "$0" + for a in "$@"; do printf 'arg %s\\n' "$a"; done + printf 'TMPDIR %s\\n' "$TMPDIR" + printf 'PYTHONPYCACHEPREFIX %s\\n' "$PYTHONPYCACHEPREFIX" + printf 'pid %s\\n' "$$" +} > "$FAKE_SUITE_LOG" +: > "$TMPDIR/the-suite-wrote-here" +if [ -n "${FAKE_SUITE_READ:-}" ]; then + # A DEBUGGER'S PROMPT: it reads the terminal, as `--pdb` would. + printf 'prompt>\n' + read -r line + printf 'got: %s\n' "$line" + exit 0 +fi +if [ -n "${FAKE_SUITE_SLEEP:-}" ]; then + # A CHILD OF ITS OWN, as pytest waits on a shell suite or a `git`: the + # wrapper must stop it too, not only the process it started. + sleep "$FAKE_SUITE_SLEEP" & + printf '%s\n' "$!" > "$FAKE_SUITE_LOG.child" + : > "$FAKE_SUITE_LOG.started" + wait + exit 0 +fi +exit "${FAKE_SUITE_RC:-0}" +""" + +FAKE_PGREP = "#!/bin/sh\nexit 1\n" + + +class Box: + def __init__(self, root: Path, flock: bool = True): + self.root = root + self.fakebin = root / "fakebin" + self.fakebin.mkdir() + self.lockdir = root / "locktmp" + self.lockdir.mkdir() + self.state = root / "state" + self.cache = root / "cache" + self.log = root / "suite.log" + for name, text in (("python3", FAKE_SUITE), ("pgrep", FAKE_PGREP)): + (self.fakebin / name).write_text(text) + (self.fakebin / name).chmod(0o755) + if flock: + path = f"{self.fakebin}{os.pathsep}{os.environ.get('PATH', '')}" + else: + # NO `flock` ON THIS PATH: the wrapper takes its `mkdir` lock, the + # one macOS uses. Every other tool it calls is linked in by name. + for tool in ("dirname", "awk", "mkdir", "cat", "mv", "rm", "sleep", "date", + "chmod", "ps"): + found = shutil.which(tool) + assert found, tool + (self.fakebin / tool).symlink_to(found) + path = str(self.fakebin) + self.env = {k: v for k, v in os.environ.items() + if not k.startswith(("XDG_", "PYTHON", "FAKE_"))} + self.env.update(PATH=path, TMPDIR=str(self.lockdir), XDG_STATE_HOME=str(self.state), + XDG_CACHE_HOME=str(self.cache), HOME=str(root / "home"), + FAKE_SUITE_LOG=str(self.log)) + + def popen(self, *args, **env) -> subprocess.Popen: + """The wrapper, its output to FILES: a pipe is held open by every + process that inherited it, and `communicate` waits for all of them.""" + child = dict(self.env) + child.update(env) + self.out = open(self.root / "wrapper.out", "w") + self.err = open(self.root / "wrapper.err", "w") + return subprocess.Popen([shutil.which("bash") or "bash", str(WRAPPER), *args], + env=child, stdout=self.out, stderr=self.err, + stdin=subprocess.DEVNULL) + + def finish(self, proc: subprocess.Popen, timeout: float = 60) -> str: + proc.wait(timeout=timeout) + self.out.close() + self.err.close() + return (self.root / "wrapper.err").read_text() + + def recorded(self) -> dict: + out: dict = {"arg": []} + for line in self.log.read_text().splitlines(): + key, _, value = line.partition(" ") + if key == "arg": + out["arg"].append(value) + else: + out[key] = value + return out + + def runs(self) -> list: + base = self.state / "openRepoTools" / "tmp" + return sorted(base.iterdir()) if base.is_dir() else [] + + def wait_started(self, timeout: float = 30) -> None: + deadline = time.time() + timeout + marker = Path(str(self.log) + ".started") + while time.time() < deadline: + if marker.exists(): + return + time.sleep(0.1) + raise AssertionError("the stub suite never started") + + +def test_a_run_relocates_bytecode_caches_and_temp_and_leaves_none_of_it(tmp_path): + box = Box(tmp_path) + proc = box.popen("-k", "lane_worktrees", FAKE_SUITE_RC="5") + err = box.finish(proc) + assert proc.returncode == 5, err + rec = box.recorded() + run_root = Path(rec["TMPDIR"]).parent + assert run_root.parent == box.state / "openRepoTools" / "tmp" + assert run_root.name.endswith(f"-{proc.pid}"), run_root.name + assert rec["PYTHONPYCACHEPREFIX"] == str(box.cache / "openRepoTools" / "pycache") + assert rec["arg"][:5] == ["-m", "pytest", "tests", "-q", "-p"], rec["arg"] + assert "no:cacheprovider" in rec["arg"] + assert f"--basetemp={run_root}/basetemp" in rec["arg"] + assert rec["arg"][-2:] == ["-k", "lane_worktrees"] + assert not run_root.exists(), "the run root outlived the run" + assert box.runs() == [] + # THE LOCK IS THE WORKSTATION'S, under the caller's TMPDIR and not the + # run's: the `flock` file where there is `flock`, and on macOS, which has + # none, the `mkdir` lock beside it - released by the exit. + if shutil.which("flock"): + assert (box.lockdir / "openrepotools-pytest.lock").exists(), ( + "the lock is the workstation's, under the caller's TMPDIR, not the run's") + else: + assert not (box.lockdir / "openrepotools-pytest.lock.d").exists(), "the lock was kept" + + +@pytest.mark.parametrize("sig,rc", [(signal.SIGTERM, 143), (signal.SIGINT, 130)]) +@pytest.mark.parametrize("flock", [True, False], ids=["flock", "mkdir-lock"]) +def test_a_killed_run_removes_its_run_root_and_stops_the_suite(tmp_path, sig, rc, flock): + """THE TRAP, PROVED: the wrapper is signalled while the suite is running. + It stops the suite, removes the run root (and, on the `mkdir` path, the + lock), and exits with the signal's code - not after the suite would have + finished, which is what a foreground child would have made it do.""" + if flock and shutil.which("flock") is None: + pytest.skip("this host has no flock; the mkdir-lock case covers it") + box = Box(tmp_path, flock=flock) + proc = box.popen(FAKE_SUITE_SLEEP="120") + try: + box.wait_started() + rec = box.recorded() + run_root = Path(rec["TMPDIR"]).parent + assert (run_root / "tmp" / "the-suite-wrote-here").exists() + if not flock: + assert (box.lockdir / "openrepotools-pytest.lock.d").is_dir() + started = time.time() + proc.send_signal(sig) + box.finish(proc, timeout=30) + finally: + if proc.poll() is None: + proc.kill() + proc.wait() + assert proc.returncode == rc + assert time.time() - started < 20, "the trap waited for the suite" + assert not run_root.exists(), "the run root outlived a killed run" + child = Path(str(box.log) + ".child").read_text().strip() + for pid, what in ((int(rec["pid"]), "the suite"), (int(child), "the suite's own child")): + deadline = time.time() + 10 + while time.time() < deadline and _running(pid): + time.sleep(0.1) + if _running(pid): + os.kill(pid, signal.SIGKILL) + raise AssertionError(f"{what} outlived its wrapper") + if not flock: + assert not (box.lockdir / "openrepotools-pytest.lock.d").exists(), "the lock was kept" + + +@pytest.mark.skipif(not hasattr(os, "openpty") or sys.platform.startswith("win"), + reason="needs a pseudo-terminal") +def test_a_suite_that_reads_the_terminal_is_not_stopped(tmp_path): + """Adversarial review B7: with a terminal on stdin the suite runs in the + foreground, in the terminal's process group - a debugger's prompt reads + it. A background group would be stopped by SIGTTIN and the wrapper would + wait on it for ever, holding the workstation's lock.""" + import pty + import select + box = Box(tmp_path) + env = dict(box.env, FAKE_SUITE_READ="1") + pid, master = pty.fork() + if pid == 0: # the child: a session on the pty + try: + os.execve(shutil.which("bash") or "/bin/bash", ["bash", str(WRAPPER)], env) + finally: + os._exit(127) + seen, sent, status, finished = b"", False, None, False + deadline = time.time() + 30 + try: + while time.time() < deadline: + ready, _w, _x = select.select([master], [], [], 0.2) + if ready: + try: + chunk = os.read(master, 4096) + except OSError: + chunk = b"" + seen += chunk + if b"prompt>" in seen and not sent: + os.write(master, b"hello\n") + sent = True + done, status = os.waitpid(pid, os.WNOHANG) + if done: + finished = True + break + else: + raise AssertionError(f"the wrapper hung on a suite reading the terminal: {seen!r}") + finally: + if not finished: + try: + os.killpg(pid, signal.SIGKILL) + os.waitpid(pid, 0) + except OSError: + pass + os.close(master) + assert b"got: hello" in seen, seen + assert os.WIFEXITED(status) and os.WEXITSTATUS(status) == 0, (status, seen) + assert box.runs() == [], "the run root outlived the run" + + +def _running(pid: int) -> bool: + """Alive and not a zombie: an orphan's zombie answers `kill -0` until + whoever adopted it reaps it.""" + proc = subprocess.run(["ps", "-o", "stat=", "-p", str(pid)], capture_output=True, text=True) + state = proc.stdout.strip() + return bool(state) and not state.startswith("Z") + + +def test_the_repository_venv_is_used_when_it_has_pytest(tmp_path): + box = Box(tmp_path) + venv_bin = box.cache / "openRepoTools" / "venvs" / "openRepoTools" / "bin" + venv_bin.mkdir(parents=True) + stub = venv_bin / "python3" + stub.write_text("#!/bin/sh\ncase \"$1\" in -c) exit 0 ;; esac\nexec " + f"{box.fakebin / 'python3'} \"$@\"\n") + stub.chmod(0o755) + proc = box.popen() + err = box.finish(proc) + assert proc.returncode == 0, err + assert "runs from the repository venv" in err + assert box.recorded()["arg"][:2] == ["-m", "pytest"] + assert box.runs() == [] diff --git a/tests/test_wip_init_command.py b/tests/test_wip_init_command.py index 7e32cbb..933e4c4 100644 --- a/tests/test_wip_init_command.py +++ b/tests/test_wip_init_command.py @@ -2071,3 +2071,62 @@ def test_a_commit_that_fails_prints_what_git_said(tmp_path): f"the refusal dropped what git said:\n{result.stderr}") assert not (home / ".agents" / "workspace.yaml").exists(), ( "the pointer file was written by a run that never committed") + + +# --- the workspace ignores bytecode and caches (opensoft/openRepoTools#162) -- + +#: What `openRepoTools`' `WIP_HYGIENE_LINES` adds to the seeded `.gitignore`. +HYGIENE_LINES = ("__pycache__/", "*.pyc", "*.pyo", ".pytest_cache/", ".mypy_cache/", + ".ruff_cache/", "node_modules/", ".venv/", "venv/", "site-packages/") + + +def test_the_seeded_gitignore_ignores_bytecode_caches_and_venvs(tmp_path): + """Cause 19 of the cleanup analysis: generated bytecode was committed into + handoff attachments. The seed's `.gitignore` is the template's AND the + ten hygiene lines, so `git add` of an attachment folder never picks a + `__pycache__` up; the re-run compares the same bytes and has nothing to + do.""" + home = tmp_path / "home" + env = fake_gh(tmp_path) + first = run_wip(home, extra=env) + assert first.returncode == 0, first.stdout + first.stderr + checkout = home / "projects" / "brettheap-wip" + lines = (checkout / ".gitignore").read_text(encoding="utf-8").splitlines() + template = (TEMPLATE / ".gitignore").read_text(encoding="utf-8").splitlines() + assert lines[:len(template)] == template, "the template's own lines come first, unchanged" + for line in HYGIENE_LINES: + assert lines.count(line) == 1, (line, lines) + probe = checkout / "handoffs" / "attachments" / "x" / "__pycache__" / "m.cpython-312.pyc" + probe.parent.mkdir(parents=True) + probe.write_bytes(b"\0") + ignored = subprocess.run(["git", "-C", str(checkout), "check-ignore", "-q", str(probe)], + check=False) + assert ignored.returncode == 0, "the workspace would commit bytecode" + probe.unlink() + second = run_wip(home, extra=env) + assert second.returncode == 0, second.stdout + second.stderr + assert "nothing to do" in second.stdout + + +@pytest.mark.parametrize("have,expect", [ + ("*.swp\n", HYGIENE_LINES), + ("__pycache__/\n*.pyc\nvenv/\n", tuple(x for x in HYGIENE_LINES + if x not in ("__pycache__/", "*.pyc", "venv/"))), + ("".join(f"{x}\n" for x in HYGIENE_LINES), ()), +]) +def test_a_hygiene_line_the_template_already_carries_is_not_repeated(tmp_path, have, expect): + """`wip_hygiene_lines`, by itself: the block less every line the file + already carries, and NOTHING - not even the comment - when it carries them + all, so a template that adopts the lines upstream seeds them once.""" + gitignore = tmp_path / "gitignore" + gitignore.write_text(have, encoding="utf-8") + source = COMMAND.read_text(encoding="utf-8") + start = source.index("\nWIP_HYGIENE_LINES=(") + end = source.index("\n}\n", source.index("\nwip_hygiene_lines()", start)) + 3 + proc = subprocess.run(["bash", "-c", source[start:end] + '\nwip_hygiene_lines "$1"\n', + "bash", str(gitignore)], capture_output=True, text=True, check=False) + assert proc.returncode == 0, proc.stderr + added = [ln for ln in proc.stdout.splitlines() if ln and not ln.startswith("#")] + assert tuple(added) == expect + if not expect: + assert proc.stdout == ""