Repository navigation
Description
Overview
Phase 3a of the autonomous-orchestrators epic: the deterministic watcher's core, in fake-worker mode. A stdlib tick() that honours the STOP kill switch first, dispatches nothing without an open shift brief, claims tasks through exclusive-create leases so overlapping ticks cannot double-dispatch, recovers a crashed worker at most once per shift, and enforces concurrency/session caps. Real claude --bg workers are 3b; the Mind writer 3c; the digest 3d. It also lands the AUTONOMY.md doctrine naming the watcher's network surface, which the Brain's hard invariant requires before any of it ships.
Plan
- Watcher module with a single
tick(): STOP → open-brief gate → tick lock → reap leases → plan inside the brief → caps → exclusive lease → spawn. - Fake mode only: stub worker processes; real dispatch refused until 3b.
- Crash recovery: a dead worker's lease is declared dead once, counted, and its task re-dispatched at most once per shift.
- Caps: concurrent workers and sessions per shift, as optional brief keys with defaults, documented in
batches/AGENTS.mdand validated bybrief --check. - Entry point plus systemd user unit templates and an install note; installing the timer is the human's act.
- AUTONOMY.md dated network-surface section for the watcher, pinned by a doctrine test.
Tier: judge — merge mode: human /prm
Detailed implementation plan
Affected Repositories
- PyAutoBrain (primary)
- PyAutoMind
Branch Survey
| Repository | Current Branch | Dirty? |
|---|---|---|
| ./organs/PyAutoBrain | main | clean |
| ./organs/PyAutoMind | main | clean |
Suggested branch: feature/autonomous-orchestrators-p3a (both repos); worktree root ~/Code/PyAutoLabs-wt/autonomous-orchestrators-p3a/
Implementation Steps
PyAutoBrain/agents/conductors/batch/_orchestrator.py(stdlib). State home$PYAUTO_ORCHESTRATOR_HOMEorpyauto_wt_root()/.orchestrator(no workspace path in organ code — tenant firewall); slot dir per brief stem holdingSTOP,leases/,counters.json,tick.log.tick(slot_dir, brief_path, mind, now, spawner): (1)STOPexists → no-op before reading anything else; (2)_brief.checkok andnow < review-at; (3) tick lock byO_CREAT|O_EXCLwith stale-lock TTL; (4) reap leases — exited pid with a.donemarker → released; without → dead,crashes += 1, one re-dispatch per task per shift; (5) candidates from_batch.plan(survey(mind), brief=…, session_lane="local-dev"); (6) caps → exclusive lease create → spawn. Lease JSON: task, worktree, session_id, pid, started, heartbeat, expires.- Spawners:
FakeSpawner(stub python child that sleeps then writes.done, or crashes on instruction);RealSpawnerraises "arrives in 3b".PYAUTO_ORCHESTRATOR_FAKE=1required. PyAutoBrain/bin/orchestrator_watch.py --brief <file> [--once]+bin/systemd/pyauto-orchestrator.{service,timer}templates (%h+EnvironmentFile, no workspace paths); install commands documented inskills/batch/batch.mdas the human's act._brief.py: optionalmax-workers(default 2) /max-sessions(default 6), whole numbers. PyAutoMindbatches/AGENTS.mddocuments them; the example brief carries them.PyAutoBrain/AUTONOMY.md: dated "The watcher's network surface — 2026-11-03" (opt-in, laptop-only readsgh pr checks/claude agents --json; one per-shift issue comment, 3d), referenced from the hard-invariant network bullet, with a Revert condition;tests/test_autonomy_doctrine.pypins it (revert-condition count 11 → 12).PyAutoBrain/tests/test_orchestrator_watcher.py— witness clauses (a)–(e); (a) uses two real overlapping processes. Tenant-firewall check run against the task worktree before ship.
Key Files
PyAutoBrain/agents/conductors/batch/_orchestrator.py(new),_brief.pyPyAutoBrain/bin/orchestrator_watch.py,PyAutoBrain/bin/systemd/(new)PyAutoBrain/AUTONOMY.md,PyAutoBrain/skills/batch/batch.mdPyAutoMind/batches/AGENTS.md,PyAutoMind/batches/examples/shift_brief_example.md
Original Prompt
Click to expand starting prompt
Autonomous orchestrators phase 3a: watcher core — brief gate, kill switch, leases, caps, fake workers, network-surface doctrine
Type: feature
Target: PyAutoBrain
Repos:
- PyAutoBrain
- PyAutoMind
Difficulty: medium
Autonomy: human-required
Priority: high
Status: formalised
Filed: 2026-11-03
Epic: autonomous-orchestrators
Phase: 3
Consequence: judge
Witness: a dry shift with fake workers (PYAUTO_ORCHESTRATOR_FAKE=1, tmp state dir, tmp Mind): (a) two overlapping watcher ticks dispatch exactly one worker per lease; (b) a killed worker's lease expires and the task is re-dispatched exactly once per shift, the crash recorded in the slot's counters; (c) aSTOPfile in the slot directory makes the next tick a no-op before it reads anything else; (d) the concurrent-worker and sessions-per-shift caps refuse the N+1th spawn and the tick log says why; (e) no open brief, or a brief pastreview-at:, dispatches nothing; AUTONOMY.md carries a dated network-surface section for the watcher (revert condition count +1) pinned by a doctrine test;pytest PyAutoBrain/tests -qgreen
Review-minutes: 20
Unattended: never
Lane: local-dev
Parent: draft/feature/pyautobrain/autonomous_orchestrators_epic.md — read "Design principles",
"Cross-cutting requirements" (crash recovery, usage-rate guard, Mind write contention, digest
delivery) and "External facts" (recheck them: they were verified 2026-11-03 and say recheck at
phase 3). Doctrine is already merged: PyAutoBrain AUTONOMY.md "Shift briefs and unattended
glance — 2026-11-03" and PyAutoMind policy/shift_watcher.md (the watcher is the one exemption
to end_at_deliverable; no Claude tool, arms nothing inside a session, no dispatch without an open
brief, STOP checked first). Phase 2 shipped batch brief --check / plan --brief
(agents/conductors/batch/_brief.py) — build on it.
Slicing (human decision 2026-11-03)
Phase 3 is sliced into four ordered judge-tier tasks: 3a (this prompt) watcher core;
3b real dispatch (autonomous_orchestrators_p3b_dispatch.md); 3c one Mind writer
(autonomous_orchestrators_p3c_mind_writer.md); 3d digest + delivery
(autonomous_orchestrators_p3d_digest.md). 3a owns deliverables 1, 2 and the caps half of 4
below, in fake-worker mode only, plus the AUTONOMY.md network-surface edit: the Brain's hard
invariant ("any further network surface is a new decision") must name the watcher's opt-in,
laptop-only reads (gh pr checks, claude agents --json) and the one per-shift issue comment
(posted in 3d) before any of it ships. Deliverables 3, the token half of 4, 5 and 6 belong to
3b–3d.
Deliverables (whole phase; see Slicing for 3a's share)
- The watcher — a stdlib, non-LLM script (e.g.
PyAutoBrain/bin/orchestrator_watch.py)
run by a systemd user timer under WSL. Each tick, in order:STOPin the slot directory →
exit; no open brief (valid perbrief --check,review-at:not passed) → exit; read leases,
gh pr checks, Mind state; decide; spawn or collect. Ship the unit/timer files as templates
plus an install note — installing the timer on this machine is the human's act, not the
session's (the end_at_deliverable hook stays as is for every session). - Leases —
~/Code/PyAutoLabs-wt/.orchestrator/<slot>/leases/<task>.json(task, worktree,
session id, started, heartbeat, expires), written atomically with an exclusive create so two
ticks cannot both claim a task; stale lease past expiry → declared dead once, worktree
inspected, task re-dispatched at most once per shift.worktree_check_conflictstill runs
inside the worker. batch dispatch— spawns a worker from the orchestrator's session template (sources,
bootstrap, the brief path,/start_dev <path> --auto, the end-at-deliverable contract),
asclaude --bg(fake mode: a stub process). Never bare.- Usage-rate guard — caps on concurrent workers and sessions per shift, plus the brief's
token-budget:consumed from each session's final usage report; at a cap the watcher stops
spawning and the digest says why. - One Mind writer — workers write ledger lines (claim, close-out) as files in the lease
directory; only the orchestrator commits to PyAutoMind, from a temporary worktree with
cherry-pick onto origin/main (never the canonical checkout, never reset --hard). batch digest— appends## Digestto the brief record perbatches/AGENTS.md(brief
work vs backlog picks with reasons, one proof packet per task, the counters), and posts it
once as a comment on the record'sissue:. Phase 2's validator gains a digest check.
Out of scope
The pilot (phase 4), RAL packets (phase 5), the Science orchestrator (phase 6). No merge logic
beyond what AUTONOMY.md already grants; unattended glance close-outs reuse /prm.
Notes for start_dev
Large and multi-surface: expect start_dev to slice this into ordered PRs (e.g. leases + fake
mode + kill switch first; dispatch + usage guard; Mind writer; digest + delivery), each
judge-tier.
Activity
Jammy2211 commentedon Nov 3, 2026
Shipped — merged 2026-11-03 via /prm.
max-workers/max-sessionsdocumented; example brief carries them._orchestrator.pySTOP-first tick, open-brief gate, tick lock, exclusive-create leases, crash re-dispatch once per shift, caps, fake workers (real dispatch refused until 3b);bin/orchestrator_watch.py+ systemd templates (installing them is the human's act); AUTONOMY "The watcher's network surface — 2026-11-03".Witness: 16/16 watcher tests (four racing tick processes), doctrine 14/14, revert conditions 11 → 12, tenant firewall OK against the branch, full suites Brain 1456 / Mind 712. CI green first time on both. Next: 3b (
draft/feature/pyautobrain/autonomous_orchestrators_p3b_dispatch.md) — worker permission mode to be decided with the human at plan time.