OTA Phase 4: host-driven automated rollout (desired version, manifest, guardrails)

[TalkingJupiter]

Phase 4 of the OTA foundation (#27), building on the control plane from Phase 3 (#37). Phase 3 lets the host command an update. This phase decides when to command one, so that updating the fleet requires nobody to touch a device or run anything by hand.

The model: reconcile desired against actual

The host holds a desired firmware version and compares it against what each device reports. Where they differ, it issues an update. Where they match, it does nothing. This is a reconciliation loop, not a deployment script: it converges rather than executing a one-shot sequence, so a device that was powered off during a rollout picks up its update when it comes back.

Actual state already exists. current_status.running_firmware_version is populated from every hello.

Desired state does not. Something has to record "these devices should run v0.0.3". The granularity is a real decision:

• Global is simplest and matches a homogeneous fleet, which is what Redesign dual-master architecture: independent per-controller intake/exhaust buses + unified firmware #26's unified image gives us.
• Per rack allows staging a rack at a time and is the natural unit for the rolling update in Phase 5.
• Per role is probably wrong: Primary and Standby run the same image by design, and diverging them undermines failover.

Recommendation: a single desired version with an optional per-rack override, which covers canary and staged rollout without inventing a policy engine.

Component 1: the firmware manifest

fw-server currently serves files and nothing else, and the SHA-256 is computed by hand. Automation needs a machine-readable manifest generated when a binary is published:

{
  "version": "v0.0.3",
  "file": "repacss-fw-v0.0.3.bin",
  "size": 360672,
  "sha256": "884ce0f6...",
  "published_at": "2026-08-03T12:44:10Z"
}

This is the artifact that removes the human from the loop. Whether it is a static manifest.json regenerated on publish, or a small service, is an implementation detail; the requirement is that the updater never needs a person to supply a hash.

Component 2: the reconciler

A service that periodically compares desired against actual and issues Phase 3 commands for the difference. Needs to be idempotent in its own right: it should tolerate running every minute and not re-command a device whose update is already in flight. Phase 3's command_id and ota/status reports are what make that possible, so the reconciler tracks in-flight commands rather than firing blindly on version mismatch.

Component 3: guardrails

This is the part that matters most, because automation without it means one bad image reaches every controller in the building on a timer. The probation rollback from #31 means a bad image self-heals rather than bricking, which is a strong safety net, but it is not a substitute for not shipping it fleet-wide in the first place.

• Never update both controllers in a rack at once. The host knows roles from device_map; the firmware knows peerAlive. Both sides should enforce it, since either alone can be wrong.
• Canary. Update one device, wait for booted plus a healthy interval, then proceed. A device that reports booted and then goes quiet 90 seconds later has not passed.
• Halt on failure. After N failed or reverted updates, stop and raise an alert rather than continuing down the fleet. The alerts table already exists.
• Kill switch. A way to stop all updates without redeploying a container. A row in the database is enough; the point is that stopping does not require a deploy.
• Quiet the target first. enabled:false (Firmware: accept enabled:false to support remote disable #19) exists precisely so a device can be told to stand down before it reboots, letting its partner cover the rack. This is where that finally earns its keep, and it is blocked on Firmware: accept enabled:false to support remote disable #19 being implemented, since the firmware currently rejects a disabled config outright.

Open questions

1. How is the desired version set? A database row edited by hand is fine to start and honest about what it is. A dashboard is Phase 5+ territory.
2. What counts as "healthy" for the canary? Reaching MQTT is the weakest useful signal and is already covered by probation. Publishing valid telemetry for a couple of intervals is stronger and catches the healthy-but-misbehaving case that OTA: firmware support for host-commanded rollback (healthy-but-misbehaving image) #32 exists for.
3. Automatic rollback on canary failure? OTA: host canary health-gate + commanded rollback on bad firmware #33 covers the host-side health gate. Worth deciding whether Phase 4 halts and alerts, or halts and reverts. Halting is safer to build first.

Acceptance criteria

• Setting a desired version causes every device below it to update, with no human action beyond setting the version.
• A rack never has both controllers updating or rebooting at the same time.
• A failed canary halts the rollout and raises an alert instead of continuing.
• A device that was offline during a rollout updates on its own when it returns.
• The kill switch stops further updates without a redeploy.

Part of #27. Builds on #37. Blocked on #19 for quieting a target. Relates to #32 and #33 for the misbehaving-image case.