Skip to content

feat: autonomous orchestrators p3a — watcher core (fake workers, leases, kill switch) #555

Closed

Description

@Jammy2211

Overview

Phase 3a of the autonomous-orchestrators epic: the deterministic watcher's core, in fake-worker mode. A stdlib tick() that honours the STOP kill switch first, dispatches nothing without an open shift brief, claims tasks through exclusive-create leases so overlapping ticks cannot double-dispatch, recovers a crashed worker at most once per shift, and enforces concurrency/session caps. Real claude --bg workers are 3b; the Mind writer 3c; the digest 3d. It also lands the AUTONOMY.md doctrine naming the watcher's network surface, which the Brain's hard invariant requires before any of it ships.

Plan

  • Watcher module with a single tick(): STOP → open-brief gate → tick lock → reap leases → plan inside the brief → caps → exclusive lease → spawn.
  • Fake mode only: stub worker processes; real dispatch refused until 3b.
  • Crash recovery: a dead worker's lease is declared dead once, counted, and its task re-dispatched at most once per shift.
  • Caps: concurrent workers and sessions per shift, as optional brief keys with defaults, documented in batches/AGENTS.md and validated by brief --check.
  • Entry point plus systemd user unit templates and an install note; installing the timer is the human's act.
  • AUTONOMY.md dated network-surface section for the watcher, pinned by a doctrine test.

Tier: judge — merge mode: human /prm

Detailed implementation plan

Affected Repositories

  • PyAutoBrain (primary)
  • PyAutoMind

Branch Survey

Repository Current Branch Dirty?
./organs/PyAutoBrain main clean
./organs/PyAutoMind main clean

Suggested branch: feature/autonomous-orchestrators-p3a (both repos); worktree root ~/Code/PyAutoLabs-wt/autonomous-orchestrators-p3a/

Implementation Steps

  1. PyAutoBrain/agents/conductors/batch/_orchestrator.py (stdlib). State home $PYAUTO_ORCHESTRATOR_HOME or pyauto_wt_root()/.orchestrator (no workspace path in organ code — tenant firewall); slot dir per brief stem holding STOP, leases/, counters.json, tick.log. tick(slot_dir, brief_path, mind, now, spawner): (1) STOP exists → no-op before reading anything else; (2) _brief.check ok and now < review-at; (3) tick lock by O_CREAT|O_EXCL with stale-lock TTL; (4) reap leases — exited pid with a .done marker → released; without → dead, crashes += 1, one re-dispatch per task per shift; (5) candidates from _batch.plan(survey(mind), brief=…, session_lane="local-dev"); (6) caps → exclusive lease create → spawn. Lease JSON: task, worktree, session_id, pid, started, heartbeat, expires.
  2. Spawners: FakeSpawner (stub python child that sleeps then writes .done, or crashes on instruction); RealSpawner raises "arrives in 3b". PYAUTO_ORCHESTRATOR_FAKE=1 required.
  3. PyAutoBrain/bin/orchestrator_watch.py --brief <file> [--once] + bin/systemd/pyauto-orchestrator.{service,timer} templates (%h + EnvironmentFile, no workspace paths); install commands documented in skills/batch/batch.md as the human's act.
  4. _brief.py: optional max-workers (default 2) / max-sessions (default 6), whole numbers. PyAutoMind batches/AGENTS.md documents them; the example brief carries them.
  5. PyAutoBrain/AUTONOMY.md: dated "The watcher's network surface — 2026-11-03" (opt-in, laptop-only reads gh pr checks / claude agents --json; one per-shift issue comment, 3d), referenced from the hard-invariant network bullet, with a Revert condition; tests/test_autonomy_doctrine.py pins it (revert-condition count 11 → 12).
  6. PyAutoBrain/tests/test_orchestrator_watcher.py — witness clauses (a)–(e); (a) uses two real overlapping processes. Tenant-firewall check run against the task worktree before ship.

Key Files

  • PyAutoBrain/agents/conductors/batch/_orchestrator.py (new), _brief.py
  • PyAutoBrain/bin/orchestrator_watch.py, PyAutoBrain/bin/systemd/ (new)
  • PyAutoBrain/AUTONOMY.md, PyAutoBrain/skills/batch/batch.md
  • PyAutoMind/batches/AGENTS.md, PyAutoMind/batches/examples/shift_brief_example.md

Original Prompt

Click to expand starting prompt

Autonomous orchestrators phase 3a: watcher core — brief gate, kill switch, leases, caps, fake workers, network-surface doctrine

Type: feature
Target: PyAutoBrain
Repos:

  • PyAutoBrain
  • PyAutoMind
    Difficulty: medium
    Autonomy: human-required
    Priority: high
    Status: formalised
    Filed: 2026-11-03
    Epic: autonomous-orchestrators
    Phase: 3
    Consequence: judge
    Witness: a dry shift with fake workers (PYAUTO_ORCHESTRATOR_FAKE=1, tmp state dir, tmp Mind): (a) two overlapping watcher ticks dispatch exactly one worker per lease; (b) a killed worker's lease expires and the task is re-dispatched exactly once per shift, the crash recorded in the slot's counters; (c) a STOP file in the slot directory makes the next tick a no-op before it reads anything else; (d) the concurrent-worker and sessions-per-shift caps refuse the N+1th spawn and the tick log says why; (e) no open brief, or a brief past review-at:, dispatches nothing; AUTONOMY.md carries a dated network-surface section for the watcher (revert condition count +1) pinned by a doctrine test; pytest PyAutoBrain/tests -q green
    Review-minutes: 20
    Unattended: never
    Lane: local-dev

Parent: draft/feature/pyautobrain/autonomous_orchestrators_epic.md — read "Design principles",
"Cross-cutting requirements" (crash recovery, usage-rate guard, Mind write contention, digest
delivery) and "External facts" (recheck them: they were verified 2026-11-03 and say recheck at
phase 3). Doctrine is already merged: PyAutoBrain AUTONOMY.md "Shift briefs and unattended
glance — 2026-11-03" and PyAutoMind policy/shift_watcher.md (the watcher is the one exemption
to end_at_deliverable; no Claude tool, arms nothing inside a session, no dispatch without an open
brief, STOP checked first). Phase 2 shipped batch brief --check / plan --brief
(agents/conductors/batch/_brief.py) — build on it.

Slicing (human decision 2026-11-03)

Phase 3 is sliced into four ordered judge-tier tasks: 3a (this prompt) watcher core;
3b real dispatch (autonomous_orchestrators_p3b_dispatch.md); 3c one Mind writer
(autonomous_orchestrators_p3c_mind_writer.md); 3d digest + delivery
(autonomous_orchestrators_p3d_digest.md). 3a owns deliverables 1, 2 and the caps half of 4
below, in fake-worker mode only, plus the AUTONOMY.md network-surface edit: the Brain's hard
invariant ("any further network surface is a new decision") must name the watcher's opt-in,
laptop-only reads (gh pr checks, claude agents --json) and the one per-shift issue comment
(posted in 3d) before any of it ships. Deliverables 3, the token half of 4, 5 and 6 belong to
3b–3d.

Deliverables (whole phase; see Slicing for 3a's share)

  1. The watcher — a stdlib, non-LLM script (e.g. PyAutoBrain/bin/orchestrator_watch.py)
    run by a systemd user timer under WSL. Each tick, in order: STOP in the slot directory →
    exit; no open brief (valid per brief --check, review-at: not passed) → exit; read leases,
    gh pr checks, Mind state; decide; spawn or collect. Ship the unit/timer files as templates
    plus an install note — installing the timer on this machine is the human's act, not the
    session's (the end_at_deliverable hook stays as is for every session).
  2. Leases — ~/Code/PyAutoLabs-wt/.orchestrator/<slot>/leases/<task>.json (task, worktree,
    session id, started, heartbeat, expires), written atomically with an exclusive create so two
    ticks cannot both claim a task; stale lease past expiry → declared dead once, worktree
    inspected, task re-dispatched at most once per shift. worktree_check_conflict still runs
    inside the worker.
  3. batch dispatch — spawns a worker from the orchestrator's session template (sources,
    bootstrap, the brief path, /start_dev <path> --auto, the end-at-deliverable contract),
    as claude --bg (fake mode: a stub process). Never bare.
  4. Usage-rate guard — caps on concurrent workers and sessions per shift, plus the brief's
    token-budget: consumed from each session's final usage report; at a cap the watcher stops
    spawning and the digest says why.
  5. One Mind writer — workers write ledger lines (claim, close-out) as files in the lease
    directory; only the orchestrator commits to PyAutoMind, from a temporary worktree with
    cherry-pick onto origin/main (never the canonical checkout, never reset --hard).
  6. batch digest — appends ## Digest to the brief record per batches/AGENTS.md (brief
    work vs backlog picks with reasons, one proof packet per task, the counters), and posts it
    once as a comment on the record's issue:. Phase 2's validator gains a digest check.

Out of scope

The pilot (phase 4), RAL packets (phase 5), the Science orchestrator (phase 6). No merge logic
beyond what AUTONOMY.md already grants; unattended glance close-outs reuse /prm.

Notes for start_dev

Large and multi-surface: expect start_dev to slice this into ordered PRs (e.g. leases + fake
mode + kill switch first; dispatch + usage guard; Mind writer; digest + delivery), each
judge-tier.

Activity

  1. Jammy2211 commented on Nov 3, 2026

    @Jammy2211
    ContributorAuthor

    Shipped — merged 2026-11-03 via /prm.

    Witness: 16/16 watcher tests (four racing tick processes), doctrine 14/14, revert conditions 11 → 12, tenant firewall OK against the branch, full suites Brain 1456 / Mind 712. CI green first time on both. Next: 3b (draft/feature/pyautobrain/autonomous_orchestrators_p3b_dispatch.md) — worker permission mode to be decided with the human at plan time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions