[lavaman131]

Summary

Adds docs/zero-downtime-upgrades-design.md, a design for shipping a hosted Chopin without signing anyone out, interrupting open tabs, or dropping Planner work. It records what a restart does today, with a file and line citation for each claim. It then proposes four stages that can each ship on their own. This PR changes documentation only, no code.

Key points of stage 1:

• Strictly invisible deploys. A warm standby waits on the writer lease while the old process keeps serving. The old process runs its cut in a fixed order:

  • readiness turns off for a fixed unready window (HANDOVER_UNREADY_MS, the proxy's health interval plus a margin; a process cannot count its proxies, so it never waits for a check that may not come);
  • the listener stops;
  • in-flight requests finish;
  • the lease is released;
  • sockets close with 1012.

  So the old process never refuses a routed request, and never answers after the release.

• The budget counts the browser's probe-first reconnect. On a loss, the browser sends a probe first and makes its attempt no later than RESCUE_AT (1.2 s). Adoption therefore has to happen within about 1.2 s of the loss. The worst case is about 750 ms with the reference values. The upgrade burst at 1.2 s is the thin margin, and the load test must cover it.

• Remote Planners stay attached through a deploy. Each hold whose link the cut closes is persisted with a hand-over window (proposed: 10 s). Until the window ends, browsers see the Planner as attached, and requests sent to it are accepted as durable rows. A new restart-notice client rule (retry a resume refused with retry every 250 ms) brings a client back in about 2 s when the owner has a tab open. If the window ends without a resume, members see what a lost connection shows today, and the miss is logged with its cause.

• No acknowledged Planner request is silently dropped in hosted mode. Every acknowledged request that cannot start at once becomes a planner_deferred_turns row before its acknowledgement, in normal operation as well as during a drain. Five kinds are covered: Chat @chopin messages, invoke_planner messages, system invoke_planner calls, comment notes sent to Chopin (in a root or a child), and task drafts from Build. The design lists every other queue item and why it needs no row: job turns are already durable, and server-authored turns were acknowledged to nobody.

  • A row carries what its turn needs: text or thread, references, delivery, child focus, plan revision and claimant.
  • It moves through waiting, started, ended, interrupted and cancelled. Each failure has a notice: in Chat, and also in the thread ("Chopin couldn't reply") or on Build plan.
  • A later writer's request claims the document instead of waiting behind an absent claimant.
  • Rows interrupted by a crash are reported, never replayed.

Stages and sizes

Sizes are re-estimates, not measurements.

Stage	Size	Estimate
1. Strictly invisible restarts on one server	L	14–17.5 engineer-weeks (11.5–14 without persisted Planner holds)
2. Release safety	M	4.5–5.5 engineer-weeks, plus operator tooling
3. Several processes	XL	16–26 engineer-weeks, low confidence
4. Connected machines	M	3–5 engineer-weeks

Stage 1 grew from an earlier 8–11 weeks, for these reasons:

• the ordered cut;
• a durable queue in normal operation covering five kinds of request, needed because requests queued before SIGTERM were acknowledged from memory;
• the remote-Planner hand-over window;
• crash-window, child-focus, comment-note, remote-window and proxy-in-the-loop tests.

An earlier draft of this revision said 12–15. Covering comment notes, task drafts and system calls, and adding the hand-over window, added 2–2.5 weeks. Stage 2's parts add to 4.5–5.5 weeks.

Decisions already made

• Invisible (open question 3): strictly invisible. No notice may appear within the 1.5 s grace, which requires process overlap. A missed budget falls back to today's "Reconnecting…" notice and is logged, and the release test allows no misses. The remote-Planner exceptions are bounded and listed (3 (h)). Each happens at most once per deploy per affected document, and never to a paired machine's Planner in hosted mode:
  • the owner's login cannot come back within the window, because no Chopin tab is open in the browser that holds it;
  • the agent client does not resume within the window;
  • the client never offered resumable-sessions.
• Remote Planner restart (open question 4): persist sessions. A login is still restored only from its browser cookie, so docs/authentication.md:395-396 is unchanged. An owner-missing resume gets unavailable with retry: true. The doc spells out exactly what a command-line-only owner sees: attached through the window, then disconnected, with requests waiting as rows until that browser returns.
• Platform (open questions 1 and 2): the known hosted deployment runs Docker Compose with Caddy 2 on one VM. The new reference deployment has these parts:
  • Both slots set HANDOVER=wait and HANDOVER_UNREADY_MS.
  • Per-slot pull and up --detach --no-deps steps replace the script's plain pull and up --detach. The plain up --detach recreates both slots at once after an image change.
  • Caddy uses a 250 ms health interval, a 100 ms lb_try_interval, health_timeout 1s and a 5 s hold.

Open questions for maintainers

1. What the proxy answers after its hold expires while no upstream is ready. It must be a 5xx, never a 4xx.
2. The turn-wait limit under the 30 s stop grace (open question 5).
3. Stale deferred turns (open question 15). Should a row whose sender's login is not live run on the write check made at acceptance, wait, or expire?
4. The proposed values need measuring in the restart and load tests:
   • the 100 ms lease poll;
   • the 300 ms unready window;
   • the 1 s in-flight bound;
   • the 10 s remote-Planner hand-over window;
   • the Caddy intervals, and health_timeout 1s against health_fails 2.
5. Open questions 6–14 remain open and unchanged.

Verification

• bun run ci exits 0. It ran dprint, oxlint (0 errors, 71 existing warnings), and the token and design checks.
• Citation check: 399 path:line citations and 91 :line continuations name existing files with in-range lines. I reread the code behind each new citation. I found the kinds of queued Planner turn by reading every chat.waiting.push site and every caller of Chat.instruct.
• Relative link check: all 61 relative links and anchors resolve, with exact casing.
• Caddy check with stand-ins: caddy:2 (Caddy 2.11.7) in front of two Python backends standing in for Chopin, with six clients sending mixed GET and POST requests. Each order ran 30 times, and every run's output was kept.
  • In the design's cut order (a fixed 300 ms unready window, health_timeout 1s), 30 runs gave no error in 12,702 requests. Each window answered exactly one readiness check with 503.
  • With the listener stopped while Caddy still counted the slot ready, 9 of 30 runs each had one failed request: 9 POSTs answered 502, out of 12,953 requests.
  • These runs show Caddy's behaviour only. Chopin itself was not run behind Caddy.
• caddy validate accepts the reference upstream block inside a minimal site block.
• Docker Compose 5.5.1 on a throwaway two-slot project of sleep containers sharing one image tag:
  • after the tag moved to another image, plain up --detach recreated both slots;
  • up --detach --no-deps chopin-b recreated only that slot.
• Privacy grep (case-insensitive) for private project, organization and resource names: no matches in the design doc, this PR body or the commit message.
• Not run:
  • no Chopin test (this is a documentation change);
  • no caddy validate on the operator's own site block;
  • no two-slot restart of Chopin;
  • no measurement of a real agent client's resume time.

Assistant-model: Claude Sonnet 5.5
Assistant-model: Claude Haiku 5.5
Assistant-model: Claude Opus 5.5
Assistant-workflow: goal (run 2fb55c4c-d75a-43ab-a116-6ee11a1285ac)
Assistant-verification: bun run ci passed with exit 0 (dprint, oxlint 0 errors and 71 existing warnings, token and design checks)
Assistant-verification: citation check passed: 399 path:line citations and 91 continuations in the design doc name existing files with in-range lines
Assistant-verification: relative link check passed: 61 relative links and anchors resolve with exact casing
Assistant-verification: Caddy 2.11.7 stand-in probe, 30 retained runs per order: the design's cut order gave 0 errors in 12,702 requests; stopping the listener first gave 9 failed POSTs in 12,953 requests over 9 of 30 runs
Assistant-verification: caddy validate accepted the reference upstream block in a minimal site block
Assistant-verification: Docker Compose 5.5.1 throwaway project: plain up --detach recreated both slots after an image change, and up --detach --no-deps recreated one
Assistant-verification: privacy grep passed: no private project, organization or resource names in the design doc, PR body or commit message
User-preference: Deploys must be strictly invisible to users, and Planner work must survive a server deploy
Co-authored-by: Alex Lavaee lavaman131@github.com