lane-worktrees: the daily estate report, and nothing new lands beside the code (#162, part 2)

[brettheap]

What

The second of two stacked PRs for #162 (base: feat/lane-worktrees-sweep, #168). #168 retires what is already there; this keeps more from arriving, and counts what is left every day.

1. tests/run.sh leaves nothing beside the code. PYTHONPYCACHEPREFIX=${XDG_CACHE_HOME:-~/.cache}/openRepoTools/pycache, -p no:cacheprovider, and every temporary directory of a run (--basetemp and the suite's TMPDIR) under one run root, ${XDG_STATE_HOME:-~/.local/state}/openRepoTools/tmp/<UTC>-<pid>/, which an EXIT trap removes however the run ends. The lock is computed before TMPDIR moves, so it is still the workstation's. The suite runs in its own process group, and the whole group is stopped before cleanup.
2. One virtual environment per repository, outside the estate. openRepoTools --install names ${XDG_CACHE_HOME:-~/.cache}/openRepoTools/venvs/openRepoTools, says whether it is there, and prints the two commands that make it. It never makes one (pip and the network); tests/run.sh runs from it once it has pytest.
3. lane-worktrees sweep --all --dry-run --report [--post <file|owner/repo#n>]: the estate's leftovers as one markdown document. lane-start runs it once a day per workstation, detached, guarded by a stamp file. It changes nothing.
4. The workspace never carries bytecode. wip init seeds the .gitignore lines. lanes-edit.sh's commit path refuses, exit 2 with nothing staged, a pathspec that would stage bytecode, a cache, node_modules, a venv or site-packages, and offers the missing lines (ten of them, site-packages/ included). lanes-edit.sh pathspec-check <path>... asks the same question before a hand commit of attachments.
5. Tests for each, and the docs (docs/README-lanes.md, README.md, AGENTS.md).

Why

The cleanup analysis's diagnosis is that workspace creation outpaces workspace closeout. #168 is the closeout. This PR covers the rest: things stop landing in worktrees when they are created (causes 3, 4, 19), and the remaining gap becomes a number someone reads every day (causes 7, 9, 10, 17, 20). The report catches what is left after every actor that never runs lane-end.

The report, as implemented

It is meant to read every lane's #97 inventory and snapshot under <parent>/.lane-state for every checkout's parent (#97 keeps a nested lane's control root beside its checkout). On the real estate it reads none (defect B8, deferred to #170): the estate walk never enters a plain directory inside a repository, so xFactory/xFactories/* are never found as checkouts and xFactory/xFactories/.lane-state is never read. It also reads the workspace register on origin/<branch>, and the disk. It never reads the derived index (Amendment 14 clause (b)), and that includes #161's table. It never fetches, never refreshes an index (GIT_OPTIONAL_LOCKS=0) and never expires anything.

section	lists
FOREIGN repositories	a clone inside a worktree container, and a second clone of an origin the estate already has. Each with path, size, last commit, Lane: owner, and LOAD-BEARING with its dependents
Orphaned worktrees of ENDED lanes	inventory trees and .lane-worktrees/<lane>/* of a lane whose snapshot is CLOSED or whose log ends ENDED/RETIRED
Unmerged branches	no upstream / gone / diverged / unpushed, with tip, age, owner and "held by a worktree". Never deleted
Root main divergence	local main ahead of origin. Where only handoffs//lanes/ paths moved, the Amendment 4 remedy is given
Awaiting disposition	rescue/* branches and dirty inventory trees older than aging_days (14), with owner and age
Caches and sandboxes	by size
Ignored directories over ignored_report_mb (50)	du -sk per ignored directory
Evidence-shaped paths	untracked/ignored junit*.xml, MANIFEST*, *-report.md, *REPORT*.md, canary-*, *-evidence, and a directory of reports beside the repositories. Each is a finding to move to the one evidence root, ${XDG_STATE_HOME}/openRepoTools/evidence/<repo>/<UTC>-<slug>/
Sweep archives past retention	--expire's own table (now shared as expiry_rows): what would expire, and what is kept because its rescue left origin
Workspace repository hygiene	the .gitignore lines the workspace lacks (with the command that adds them), and bytecode already in its history (a person's word)
status --all findings	pin drift, ahead/behind, forks, parked records, reported as what they are and never counted as dirt

The head table also carries the rescue-branch count and the sweeps directory's size.

Daily run (lane-start § 5e): runs after the row is written and before the launch. It takes report-<YYYYMMDD>.stamp with an atomic set -C create, so two simultaneous starts run one report. It removes older stamps, then runs the report with stdin, stdout and stderr closed, in a subshell that exits at once, writing ${XDG_STATE_HOME}/openRepoTools/reports/<UTC>.md. Every step either works or is skipped silently. --dry-run starts none, and LANE_WORKTREES_REPORT=off turns it off (the suite sets that).

Where this differs from the issue's wording, and why

• Expiry is listed, not performed. The fold says --expire is "run by the nightly report", but the brief requires a report that changes nothing. The report therefore prints --expire's dry run, and expiring stays the act sweep --expire --yes.
• The venv is named, not created. "Create or name it": an install from a checkout has no pip or network guarantee.
• Derived index: a worktrees table mirroring #97's per-lane inventory (path, branch, lifecycle, owning lane, writer liveness, dirty/unpushed, last seen) and lanes --index --worktrees — so the next account sees every open worktree in one query #161's index is not read, under Amendment 14(b).

Tests

file	cases	holds
tests/test_run_wrapper.py (new)	6	Stub suite and stub pgrep. Checks bytecode prefix, -p no:cacheprovider, --basetemp under the run root, run root removed after the run, lock still under the caller's TMPDIR. A killed run (TERM→143, INT→130, × flock / mkdir-lock) stops the suite and a child of its own within seconds, removes the run root and releases the mkdir lock. The venv is used when it has pytest
tests/test_lane_worktrees_report.py (new)	9	Builds an estate with every finding: foreign clone + duplicate clone + two clones of one origin with neither named for it, CLOSED-lane tree (top-level and nested parent), no-upstream and diverged branches, handoff-only root main, 30-day rescue branch and dirty inventory tree, cache, killed tmp.*, ignored build/, JUnit file, a reports directory, a would-expire and a kept archive, workspace .gitignore + .pyc in history, status --all (pointed at the same estate, with or without --estate). Every section finds its own, and a whole-estate snapshot is unchanged. --post to a file and to owner/repo#n (fake gh). --all/--report misuse → 64. A record the report cannot read is a finding ("Records the report could not read"), never treated as absent. lane-start: the report runs detached (a 90 s reporter, start returns before it finishes), a stamp from today runs no second report, yesterday's is replaced, a failing reporter never fails a start, off and --dry-run start none
tests/test_openrepotools_command.py	+2	--install names the venv: "not created" with the two commands, then "present". Never creates one. The printed command survives a cache path with a space
tests/test_wip_init_command.py	+4	Seeded .gitignore = template + the ten lines, each once. git check-ignore of an attachment .pyc. Re-run "nothing to do". wip_hygiene_lines repeats no line the file has, and prints nothing when it has them all (×3)
tests/test_lane_helpers.sh	+19 assertions	pathspec-check refuses an attachment with __pycache__, a venv/ and a venv under another name (names them but not the notes, and offers the lines, site-packages/ included). The other-named venv is still refused until site-packages/ is added. A clean pathspec passes. commit refuses the same pathspec with 2, commits nothing and stages nothing. With the lines in place nothing of the kind is staged, and bytecode already force-staged is still refused by both pathspec-check and the commit path

Mutation-checked: with the nested-parent lookup removed, the report case fails on the nested orphan.

Locally (Eagle, tests/run.sh was queued behind other estates' pytest runs, so these used a direct harness that calls the test functions): run-wrapper 6/6, report 7/7, wip-init module 63/63, test_repo_hygiene.py 152/152, the venv case and both install idempotence cases, #168's sweep module 23/23, and bash -n on every touched script. CI is the record.

Measurements (Eagle, read-only)

lane-worktrees sweep --all --dry-run --report --estate /workspace/projects: 68.5 s (a second run 79.5 s), exit 0, 524 lines, nothing changed.

finding	count
FOREIGN repositories	89
Orphaned worktrees of ENDED lanes	0, but this is not a reading of the estate's snapshots: the report read no .lane-state on Eagle (B8, #170). xFactory/xFactories/.lane-state exists, and by hand it holds five RUNNING lanes with empty inventories; the report never reached it, and the section said "none" where it should have said what it could not read. The one RETIRED lane in the register, browser-ui-repair, has no tree
Unmerged branches (missing / diverged / unpushed upstream)	162
Root main divergence	1 (openRepoProject, 1 commit, 103 paths: not handoff-only)
Awaiting disposition (> 14 d)	23 (rescue branches only: the dirty-inventory-tree half read no .lane-state, B8)
Caches and sandboxes	97, 425.5 MB (largest /tmp/pytest-of-brett/pytest-3507, 342 MB)
Ignored directories over 50 MB	56 (largest a .dart_tool at 4.0 GB)
Evidence-shaped paths	24
Archives past retention	0
Workspace hygiene	2: brett-wip/.gitignore lacks all nine lines, and 703 bytecode paths in its history (e.g. handoffs/codeXfactory/attachments/…/venv/lib/python3.12/site-packages/__pycache__/…pyc), which is cause 19 measured
status --all findings	8 (shape-pin drift)
rescue branches (all ages)	25
the sweeps directory	0 B

Review

Round 1 on #169 (Copilot, 4 threads), all taken:

• da7bc59: the suite's whole process group is stopped (set -m for the spawn; TERM, reap, ps-checked wait, KILL), so nothing it started outlives the run root or the lock.
• b0ea54e: site-packages/ is offered and seeded alongside the other lines, so following the offer clears the refusal.
• b422d8f: duplicate clones are reported even when none is named for the repository, and status --all reads the estate the report resolved; lane-start passes its $PROJECTS_ROOT.

Each new assertion fails against the previous head. All threads were answered and resolved.

#168's round-1 fixes came in by merge (f4f4915), with no rebase and no force-push. The report's archive section now inherits whole-archive and exact-SHA expiry.

Round 2 on #169 (3 threads, at 23d84da), all taken:

• 65f73ed: bytecode already staged is refused too;
• f5bccbd: the printed venv command is quoted;
• dc7971c: the report surfaces records it could not read.

#168's round-2 fixes came in by merge (892b02d).

Docs within their caps: 31db36f. CI at 38e090f failed only on test_repo_hygiene's line caps (AGENTS.md 329/316, README.md 491/486; Linux tests was otherwise 963 passed). The full text lives in docs/README-lanes.md. AGENTS.md is back to 316 lines and README.md to 486, and no cap is raised.

After the cap (Copilot rounds 3-5 on #169):

• 30cc63a: taken, because an unconditional flock-file assertion would have failed the macOS landing gate;
• lane-worktrees: review findings deferred from #168/#169 (incl. five --yes data-loss paths found after the two-round cap) #170 items 3 and 9-11, filed and resolved: stop_group reaps the leader before the KILL deadline; ignored evidence FILES are skipped; the report's repair path is unquoted; the daily report's directory inherits the umask.

The Opus adversarial review of 30cc63a (standing in for Codex, which is over quota): B7 taken in 0cf1051. With a terminal on stdin the suite now runs in the foreground, so --pdb/breakpoint() no longer stop it on SIGTTIN while the wrapper holds the lock; a new pty case passes here and hangs against the previous run.sh. #168's fixes came in by merge (9dd63f0):

• --yes switched off until lane-worktrees: review findings deferred from #168/#169 (incl. five --yes data-loss paths found after the two-round cap) #170;
• lane-root/trailer ownership, and a NONE snapshot with trees refused;
• surrogateescape and a catch-all keeping the exit contract;
• escaped porcelain fields.

B8 (the report reads no .lane-state on the real estate), C2, C3, E8, E9, E10 and (d) are in #170's table. The measurement rows above were corrected for B8.

Closes #162

Refs #97 #161 #163 #164 brettheap/new-workstation#48

Lane: openRepoTools-3

🤖 Generated with Claude Code

Summary by Sourcery

Prevent new workspace debris from accumulating and provide a daily, read-only report of the estate's remaining cleanup findings.

New Features:

• Add a read-only estate report for identifying worktree, repository, branch, cache, evidence, archive, and hygiene findings across all workspaces.
• Run the estate report automatically once per UTC day from lane-start, with optional file or issue posting and no impact on startup success.
• Add workspace pathspec checks that prevent commits containing bytecode, caches, dependencies, or virtual environments.
• Add repository-specific virtual-environment guidance to openRepoTools --install.
• Seed workspace .gitignore files with rules covering generated bytecode, caches, dependencies, and virtual environments.

Bug Fixes:

• Prevent test runs from leaving bytecode, pytest caches, temporary directories, or child processes in worktrees after normal or interrupted execution.
• Ensure interrupted test runs stop their entire process group and release temporary locks.
• Make report failures and unreadable records visible without treating missing data as clean.
• Handle filesystems that cannot create non-UTF-8 worktree paths without failing the test suite.

Enhancements:

• Improve test execution isolation by relocating caches and temporary files to XDG state and cache locations while preserving workstation-wide locking.
• Document the estate report, workspace hygiene rules, and test-run isolation behavior.

Documentation:

• Document the estate report, daily reporting behavior, evidence locations, and workspace hygiene protections.

Tests:

• Add coverage for report contents, immutability, posting, daily detached execution, wrapper cleanup and signal handling, virtual-environment selection, and hygiene enforcement.

[brettheap]

@codex review

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[sourcery-ai]

Sorry @brettheap, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 1 day and 21 hours by commenting @sourcery-ai review. Upgrade to get a review now.

[sourcery-ai]

Reviewer's Guide

This PR closes the remaining workspace-estate hygiene gaps by isolating test-run artifacts, adding a read-only daily estate report, enforcing workspace commit hygiene, and documenting a repository-scoped virtual-environment workflow, with extensive shell and Python coverage for cleanup, signal, nested-estate, scheduling, and immutability behavior.

Sequence diagram for the daily estate report

sequenceDiagram
    participant LS as lane-start
    participant Stamp as Daily stamp
    participant Report as lane-worktrees sweep
    participant Estate as Estate state and disk
    participant Output as Report file

    LS->>Stamp: Create report-YYYYMMDD.stamp atomically
    alt Stamp created
        LS->>Report: Start detached sweep --all --dry-run --report
        Report->>Estate: Read inventories, snapshots, registers, and disk
        Report->>Report: Set GIT_OPTIONAL_LOCKS=0
        Report->>Output: Write UTC markdown report
    else Stamp already exists
        LS-->>LS: Skip report
    end

Loading

Flow diagram for workspace commit hygiene

flowchart TD
    Input[Commit pathspec] --> DryRun[git add --dry-run]
    DryRun --> Check{Bytecode, cache, dependency tree, or venv?}
    Check -->|No| Stage[git add and commit]
    Check -->|Yes| Refuse[Exit 2; nothing staged]
    Refuse --> Offer[Print offending paths and missing .gitignore lines]
    Attachment[Hand-committed attachment] --> PathCheck[lanes-edit.sh pathspec-check]
    PathCheck --> DryRun

Loading

File-Level Changes

Change	Details	Files
Reworks the test wrapper to isolate generated artifacts and temporary files while preserving workstation-wide locking and signal cleanup.	Routes Python bytecode and pytest temporary state to XDG cache/state locations. Disables pytest cache and places suite temp directories under a removable per-run root. Runs tests from the repository-scoped cached virtual environment when it contains pytest. Adds TERM/INT handling and coverage for both flock and mkdir locking paths.	tests/run.sh tests/test_run_wrapper.py AGENTS.md docs/README-lanes.md
Adds a read-only estate-wide sweep report and schedules a detached daily report from lane startup.	Implements sweep --all --dry-run --report with validation, markdown sections, summary counts, and file or GitHub issue output. Aggregates lane state, repository registers, Git metadata, disk findings, hygiene issues, archives, and status findings without fetching, indexing, or expiring. Adds once-per-UTC-day atomic stamp coordination, detached execution, silent failure behavior, and opt-out/dry-run handling. Adds broad fixture-based tests, including nested lane-state roots, unchanged-estate verification, posting, and lane-start behavior.	lane-worktrees lane-start tests/test_lane_worktrees_report.py tests/conftest.py tests/test_lane_helpers.sh docs/README-lanes.md AGENTS.md
Prevents generated content and dependency environments from entering workspace commits.	Makes wip init seed nine hygiene patterns without duplicating template entries. Checks the paths a commit would stage and refuses bytecode, caches, dependency trees, virtual environments, and site-packages with exit 2 before staging. Adds lanes-edit.sh pathspec-check for preflight checks on manually committed attachments. Adds tests for ignore seeding, idempotence, clean and rejected pathspecs, and no-op commit behavior.	lanes-edit.sh openRepoTools tests/test_wip_init_command.py tests/test_lane_helpers.sh docs/README-lanes.md README.md
Changes installation guidance to identify, rather than create, the repository-scoped virtual environment.	Prints the XDG cache path and whether the environment is present. Prints the venv and pytest installation commands while leaving creation to the user. Tests that installation performs no venv or network-dependent setup.	openRepoTools README.md tests/test_openrepotools_command.py

Possibly linked issues

• lane-worktrees sweep: one act that archives, rescues and removes a lane's abandoned worktrees, branches, scratch and killed-suite sandboxes after an ungraceful swap — dry run first, rescue branch or bundle before any removal, never a live writer #162: PR directly implements lane-worktrees sweep: one act that archives, rescues and removes a lane's abandoned worktrees, branches, scratch and killed-suite sandboxes after an ungraceful swap — dry run first, rescue branch or bundle before any removal, never a live writer #162’s daily estate report, temporary-file relocation, venv convention, hygiene checks, and documentation.
• lane-worktrees sweep: one act that archives, rescues and removes a lane's abandoned worktrees, branches, scratch and killed-suite sandboxes after an ungraceful swap — dry run first, rescue branch or bundle before any removal, never a live writer #162: PR is explicitly lane-worktrees sweep: one act that archives, rescues and removes a lane's abandoned worktrees, branches, scratch and killed-suite sandboxes after an ungraceful swap — dry run first, rescue branch or bundle before any removal, never a live writer #162 part 2, implementing its folded prevention and estate-reporting requirements.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[Copilot]

Copilot review overview

🟡 Changes recommended

Signal cleanup can leave descendant processes running, and several report and hygiene edge cases produce incomplete or incorrect results.

Review effort: Balanced
Findings: 4

Open (4)

• Renamed origin clones are incorrectly treated as canonical · New
• Pass resolved projects root to delegated status command · New
• Virtualenv site-packages rejection lacks a matching ignore rule · New
• Terminate and wait for pytest's entire process group · New

What changed in this PR

Adds read-only estate reporting and prevents generated residue from entering worktrees or workspace commits.

Changes:

• Adds daily estate-wide cleanup reporting.
• Isolates test caches, temporary files, and virtual environments.
• Adds workspace ignore rules and commit hygiene checks.

File	Description
lane-worktrees	Implements estate reports and posting.
lane-start	Starts the daily detached report.
lanes-edit.sh	Rejects generated artifacts in commits.
openRepoTools	Seeds ignores and reports external venv status.
tests/​run.sh	Relocates and cleans test artifacts.
tests/​conftest.py	Disables reports during tests.
tests/​test_lane_worktrees_report.py	Tests report generation and scheduling.
tests/​test_run_wrapper.py	Tests wrapper isolation and signals.
tests/​test_wip_init_command.py	Tests workspace ignore rules.
tests/​test_openrepotools_command.py	Tests venv installation guidance.
tests/​test_lane_helpers.sh	Tests pathspec hygiene checks.
README.md	Documents the repository venv.
docs/​README-lanes.md	Documents reporting and hygiene behavior.
AGENTS.md	Updates test-running guidance.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Copilot review overview

🟡 Changes recommended

Pre-staged artifacts can bypass the commit guard, and several report paths can produce incomplete or mismatched results.

Review effort: Balanced
Findings: 1 · 6

Open (7)

• Pre-staged prohibited files bypass validation · New
• Unreadable sidecars are falsely treated as empty · New
• Generated setup command fails to quote cache paths · New
• Terminate and wait for pytest's entire process group
• Virtualenv site-packages rejection lacks a matching ignore rule
• Pass resolved projects root to delegated status command
• Renamed origin clones are incorrectly treated as canonical

[Copilot]

Copilot review overview

🟡 Changes recommended

Log-read failures can produce a falsely clean report, and one generated repair command mishandles workspace paths requiring quoting.

Review effort: Balanced
Findings: 2

Open (2)

• Report unreadable remote refs instead of treating them as empty · New
• Quote workspace paths in the offered repair command · New

[Copilot]

Copilot review overview

🟡 Changes recommended

Caller-supplied pytest arguments can override the enforced temporary directory and bypass cleanup.

Review effort: Balanced
Findings: 1

Open (1)

• Forwarded pytest args can override enforced --basetemp · New

Resolved since last review (2)

• Quote workspace paths in the offered repair command
• Report unreadable remote refs instead of treating them as empty

[Copilot]

Copilot review overview

🟡 Changes recommended

Report encoding, pathspec failure handling, and terminal process cleanup contain correctness gaps.

Review effort: Balanced
Findings: 5 · 1

Open (6)

• Preserve non-UTF-8 bytes when reading .gitignore · New
• Use surrogateescape when writing the temporary report body · New
• Use surrogateescape for local report output · New
• Propagate Git inspection failures in pathspec-check · New
• Retain process-group state for terminal cleanup · New
• Document per-checkout-parent state locations · New

Resolved since last review (1)

• Forwarded pytest args can override enforced --basetemp

[sonarqubecloud]

Quality Gate passed

Issues
20 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud