Over-the-air (OTA) firmware update over W5500 Ethernet, MQTT-triggered

[TalkingJupiter]

Context

Firmware is flashed over USB today (platformio.ini sets upload_port per controller). With
controllers mounted in racks, physical access for reflashing doesn't scale, and #26 unifies the
image so every device runs the same binary — making a fleet-wide OTA path both feasible and
necessary. The MQTT contract already anticipates this: enabled:false is documented as "remote
off, e.g. OTA," and #19 tracks quieting a device during an update.

Problem

• No remote update mechanism exists — every firmware change requires USB access to each ESP32.
• Transport is wired W5500 Ethernet, not WiFi, so the standard mDNS-based ArduinoOTA /
  espota flow does not apply. OTA must work over the Ethernet client.
• The two controllers form an A/B failover pair; updating the active one must not blind the rack.

Proposal

Pull-based HTTP OTA using the Arduino Update.h API over the existing W5500 EthernetClient.
The host (Radxa) serves the firmware binary; an MQTT command tells a target device to fetch and
apply it, then reboot. This reuses the broker + provisioning channel we already have rather than
introducing a new push protocol.

Decisions needed

1. Trigger & command topic — e.g. repacss/devices/<mac>/ota carrying firmware URL + version
   • expected checksum. Per-device vs rack-wide vs broadcast.
2. Binary source — Radxa HTTP endpoint serving the .bin; where versions live and how a
   device knows its current vs available version.
3. Integrity & safety — checksum/signature verification before Update.end(); refuse on
   mismatch. Consider signed images given the control-channel spoofing concern raised in Prod network design: isolate rack telemetry segment + broker bootstrap (DHCP) #24.
4. Failover coordination — update standby first, verify healthy, then active; or set the
   target enabled:false (Firmware: accept enabled:false to support remote disable #19) so its peer takes over before it reboots. Never update both at once.
5. Rollback — behavior on failed boot after flash (bad image, power loss mid-write). ESP32
   rollback / OTA-data partition strategy.
6. Partition scheme — confirm the build uses a dual-app OTA partition table (default single-
   app layout has no OTA slot); update platformio.ini board_build.partitions if needed.
7. Ack/status — device reports OTA progress/result (downloading → verifying → applied → booted,
   or failed+reason) back over MQTT so the host knows the outcome (relates to the Publish config_ack after applying config #13 config_ack pattern).

Considerations

• OTA is a control channel: a spoofed OTA command could push malicious firmware. Pairs with broker
  auth/TLS and image signing (Prod network design: isolate rack telemetry segment + broker bootstrap (DHCP) #24 discovery-security thread).
• Must not disrupt the surviving controller's telemetry during a rolling update.
• Ethernet-only means no espota; verify Update.h + EthernetClient streaming fits in RAM/flash.

Acceptance criteria

• A device fetches and applies a new image over W5500 Ethernet, triggered by an MQTT command, and
  reboots into it.
• Image integrity is verified before commit; a corrupt/mismatched image is rejected without bricking.
• A failed flash rolls back to the previous working image.
• Rolling A/B update keeps the rack reporting throughout (peer covers the device being updated).
• OTA outcome (success/failure + reason) is reported back to the host.

Files affected

• ESP32-Firmware/platformio.ini — OTA partition table (board_build.partitions), build/version metadata
• ESP32-Firmware/src/TelemetrySender.cpp — MQTT command handling for the OTA topic
• new ESP32-Firmware/src/ OTA module (fetch + Update.h apply + verify + status)
• ESP32-Firmware/src/main.cpp — coordinate OTA with failover / enabled gating
• docs/mqtt-contract.md — define the OTA command/status topics + payload schema
• host side — firmware-serving endpoint + OTA trigger (provisioning service)

Relates to #19 (remote disable during OTA), #24 (control-channel auth/signing), #13 (ack pattern),
#26 (unified image).

[TalkingJupiter]

Design + phased plan (post-v0.0.2)

Readiness check after v0.0.2: the hard prerequisites are already in place — W5500 transport works end-to-end, the MQTT command channel exists (config subscribe), and the build uses the Arduino default.csv partition table (app0 + app1 + otadata), so OTA slots are available with no repartition. Update.h ships with the core.

Splitting this issue into two phases so we don't block a basic OTA on unproven failover:

Phase 1 — single-device, verified OTA (buildable now)

New OtaUpdater (OtaUpdater.h/.cpp), run off the MQTT callback stack:

• begin() marks the running image valid on a healthy boot (esp_ota_mark_app_valid_cancel_rollback) — the rollback anchor.
• MQTT command handler stores a pending request only (no long work in the callback).
• tick() from loop() runs the blocking update: HTTP GET the .bin via EthernetClient → stream into Update.write() while computing SHA-256 → verify hash before Update.end() (mismatch ⇒ Update.abort(), report failed, no reboot) → Update.end(true) → publish applied → ESP.restart().

MQTT contract additions:

• Command topic repacss/devices/<mac>/ota: { "message_type":"ota", "version", "url", "sha256" }
• Status topic repacss/devices/<mac>/ota/status: downloading → verifying → applied → booted, or failed + reason. Device republishes booted + running version on next boot for host-side confirmation.

Firmware prerequisites:

• Add a compile-time FW_VERSION reported in hello/status so the host can compare running vs available.
• Confirm ESP32 app-rollback support is enabled in the build (slots exist via default.csv; verify the rollback sdkconfig).

Radxa side (minimal): serve the .bin over HTTP on the telemetry IP; a trigger script computes the SHA-256, hosts the file, and publishes the ota command (later folded into the provisioning service).

Phase 2 — rack-aware rolling OTA (blocked on #3)

Orchestrated rolling update: standby first → verify healthy → primary; peer covers the reboot via heartbeat; never both at once. This depends on two-board failover being hardware-validated, so it waits on #3.

Non-negotiables

• SHA-256 verify before commit, even on the isolated LAN — the broker is allow_anonymous/no-TLS today, so any client can publish an ota command; the hash check is the guard against a corrupt/spoofed image bricking a device. Full broker auth/signing tracked under Prod network design: isolate rack telemetry segment + broker bootstrap (DHCP) #24.
• Deliberately test link-loss / power-loss mid-download (Update.h aborts safely; rollback covers a bad boot).

[TalkingJupiter]

Phased plan and locked decisions

Splitting OTA into independently testable phases, each filed as its own issue as we reach it. Foundation first, security and failover explicitly sequenced.

Phases

• Phase 1: version identity (git-tag) + rollback safety net -> OTA Phase 1: firmware version identity (git-tag) + rollback safety net #29 (filed, buildable now)
• Phase 2: core OTA mechanism (single device, verified): OtaUpdater = HTTP GET via EthernetClient, stream into Update.h, SHA-256 verify before commit, reboot. Minimal trigger, not the full control plane. Radxa serves the .bin over HTTP.
• Phase 3: MQTT control plane: repacss/devices/<mac>/ota command + repacss/devices/<mac>/ota/status state machine; host trigger computes SHA-256, hosts the file, publishes, correlates the result. Idempotency + concurrency guard.
• Phase 4: host/registry-managed rollout (still one device at a time): target version per device/rack, semver comparison, sequential batch trigger.
• Phase 5: rack-aware rolling update (blocked on Hardware-test MQTT telemetry publish path #3): standby first, verify healthy, then primary; peer covers the reboot via heartbeat; never both at once.
• Security track (parallel, must land before leaving the isolated LAN, ties to Prod network design: isolate rack telemetry segment + broker bootstrap (DHCP) #24): broker auth + image signing (authenticity, not just SHA-256 integrity).

Locked decisions

1. Version source: auto from git tag (git describe), injected at build time.
2. Version reporting: hello only (sent every boot while unconfigured; post-reboot hello doubles as the OTA-success signal).
3. Firmware hosting: its own service on the Radxa (fw-server), separate from provisioning; stable URL shape http://<radxa>:<port>/fw/<version>.bin.
4. OtaUpdater execution: MQTT callback only sets a pending flag; the blocking download runs in loop(). No dedicated task.
5. Transport: plain HTTP + SHA-256 now; authenticity via image signing later, not TLS.
6. Command retention: non-retained + version-based idempotency (a retained command would re-fire on reconnect and loop).
7. Update model: push (host decides and commands specific devices; device only reports its running version).
8. Rollback anchor: mark the image valid only after MQTT connect succeeds.
9. Retry policy: device reports failure and stays put; the host owns retries and backoff.