fix(opcua): run a fault manager in the OpenPLC tank demo

src/ros2_medkit_plugins/ros2_medkit_opcua/CHANGELOG.rst

@@ -10,6 +10,7 @@ Forthcoming
 * ``PLC_COMMS_LOST`` is cleared on every successful connect, including the first, so a fault ``fault_manager`` persisted before a restart does not stand against a healthy link. That clear, and the per-entity ``DELETE /{entity}/faults/{code}`` route when it is served by this plugin, set ``skip_correlation_auto_clear``: neither is an operator resolving a root cause, so neither may cascade-clear correlated symptom faults reported by apps in other entities. A clear reported by the device itself still cascades
 * Fault dispatches buffered while ``fault_manager`` is unreachable no longer lose one-shot events. Only the link-state ``PLC_COMMS_LOST`` clear is re-derivable (the next reconnect sends it again), so it is what the bounded buffer gives up first. Alarm reports, a device alarm's inactive edge and an operator's scoped clear age out oldest-first as before, and at most one clear per fault code is pending at a time
 * Gateway-side changes that land on this plugin's entities: a freeze-frame captured from a plugin entity now names the path that read the values in ``x-medkit.source`` (``plugin_data_provider`` or ``plugin_x_plc_data_route``, the only provenance an entity frame carries since it has no ROS topic), and the gateway no longer lists its own in-process helper nodes among the discovered apps
+* The OpenPLC tank demo (``docker/scripts/start.sh``) now runs a ``fault_manager_node`` beside the gateway, so an alarm the PLC raises becomes a fault the SOVD API serves. Without one ``GET /api/v1/faults`` answered 503 and the per-entity list came back empty, so the demo could show live PLC values and never a fault. Start-up waits for ``/fault_manager/report_fault`` to be advertised and fails with a clear message when it never appears, and ``faults.db`` is written to the same named volume as the entity freeze frames, so a fault and the frame it carries both outlive ``scripts/stop.sh``
 * Contributors: @bburda
 
 0.7.0 (2026-08-27)

src/ros2_medkit_plugins/ros2_medkit_opcua/README.md

@@ -862,22 +862,38 @@ The plugin ships a self-contained OpenPLC tank demo in `docker/` that exercises
 ```bash
 cd src/ros2_medkit_plugins/ros2_medkit_opcua/docker
 
-# Start OpenPLC + gateway (builds everything)
+# Start OpenPLC + gateway + fault manager (builds everything)
 bash scripts/start.sh
 
 # Manual testing
 curl -s http://localhost:8080/api/v1/apps/tank_process/x-plc-data | jq .
 
+# The faults the PLC's alarms raise
+curl -s http://localhost:8080/api/v1/apps/tank_process/faults | jq .
+
 # Automated tests (16 assertions)
 bash scripts/run_integration_tests.sh
 
 # Stop (keeps the gateway's state)
 bash scripts/stop.sh
 ```
 
+`start.sh` runs a `fault_manager_node` inside the gateway container, on the same
+ROS domain, started before `gateway_node` and waited on until
+`/fault_manager/report_fault` is advertised. That is the other half of the
+"Alarm-to-Fault Bridge" below: the plugin detects the alarm, the fault manager
+is what holds the fault. Without one `GET /api/v1/faults` answers 503 and the
+per-entity list comes back empty, so the demo can show live PLC values but
+never a fault. Its log is inside the container:
+
+```bash
+docker exec gateway cat /var/lib/ros2_medkit/fault_manager.log
+```
+
 `start.sh` mounts the named volume `ros2-medkit-opcua-state` at
 `/var/lib/ros2_medkit`, so the gateway's state survives `stop.sh` and the next
-`start.sh`: the entity freeze frames, `faults.db` and the rosbags. That is what
+`start.sh`: the entity freeze frames, `faults.db`, and the rosbags when
+black-box capture is enabled (it is opt-in and off by default). That is what
 lets a fault raised before the stop still serve the values frozen when it
 confirmed, with its original `captured_at` and no `x-medkit.capture_origin`,
 rather than a fresh read of the PLC as it is after the restart. `stop.sh`

src/ros2_medkit_plugins/ros2_medkit_opcua/docker/gateway_params.yaml

@@ -17,9 +17,16 @@ ros2_medkit_gateway:
     # confirmed instead of re-reading the PLC as it is now. scripts/start.sh
     # mounts the named volume "ros2-medkit-opcua-state" here, which is what
     # carries the frames across scripts/stop.sh too: that script removes the
-    # container, so the writable layer would not survive it. The test scripts
-    # start their own containers without a volume and keep the frames inside
-    # the container, which is all a single-run suite needs.
+    # container, so the writable layer would not survive it. The fault manager
+    # scripts/start.sh runs beside the gateway writes its "faults.db" into the
+    # same directory, so the fault and the frame it points at cross a restart
+    # together. A frame without its fault is unreachable, the fault detail
+    # route is what serves it. The test scripts start their own containers
+    # without a volume, so the frames stay inside the container and go with it.
+    # run_integration_tests.sh and test_all.sh run no fault manager, so there is
+    # no "faults.db" there at all, and the suites that do start one
+    # (run_alarm_tests.sh, run_discovery_race_test.sh) leave it in the container
+    # too. That is all a single-run suite needs.
     entity_freeze_frame:
       storage:
         path: "/var/lib/ros2_medkit/entity_freeze_frames.db"

src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/start.sh

@@ -3,13 +3,21 @@
 # Usage: from the ros2_medkit repo root, run
 #     bash src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/start.sh
 #
+# The gateway container also runs a fault_manager_node, so an alarm the PLC
+# raises becomes a fault the SOVD API serves instead of a 503.
+#
 # The gateway's state (entity freeze frames, faults.db, rosbags) is kept on the
 # named volume below, so it survives stop.sh and a later start.sh. Purge it with
 #     docker volume rm ros2-medkit-opcua-state
 set -eo pipefail
 
 STATE_VOLUME="${OPCUA_DEMO_STATE_VOLUME:-ros2-medkit-opcua-state}"
 
+# Printed by the gateway container once fault_manager_node has advertised its
+# services, and grepped for below. The container is where the wait happens, this
+# is how the host learns the outcome.
+FM_READY_MARKER="fault_manager ready: /fault_manager/report_fault"
+
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 DOCKER_DIR="$(dirname "$SCRIPT_DIR")"
 PLUGIN_DIR="$(dirname "$DOCKER_DIR")"
@@ -56,6 +64,42 @@ docker run -d --name gateway --network plc-demo -p 8080:8080 \
       mkdir -p /var/lib/ros2_medkit/rosbags /config
       echo 'manifest_version: \"1.0\"' > /config/manifest.yaml
       source /opt/ros/jazzy/setup.bash && source /root/ws/install/setup.bash
+      # A fault manager runs beside the gateway, in the same container and on
+      # the same ROS domain. Without one the alarms the OPC-UA plugin detects
+      # have nowhere to go: /api/v1/faults answers 503 and the demo can show
+      # live PLC values but never a fault. It is started before gateway_node so
+      # its services are advertised before the plugin calls
+      # /fault_manager/report_fault.
+      #
+      # database_path is passed explicitly rather than left to the default: it
+      # has to land on /var/lib/ros2_medkit, the named volume, so faults.db
+      # outlives the container the way the entity freeze frames already do.
+      # Rosbag capture is opt-in and stays off here. storage_path names the
+      # same volume, so a recording lands there whenever it is switched on.
+      ros2 run ros2_medkit_fault_manager fault_manager_node --ros-args \
+        -p database_path:=/var/lib/ros2_medkit/faults.db \
+        -p snapshots.rosbag.storage_path:=/var/lib/ros2_medkit/rosbags \
+        > /var/lib/ros2_medkit/fault_manager.log 2>&1 &
+      # Poll for the service instead of sleeping a fixed time: 'ros2 service
+      # list' is the cheapest ROS-native availability signal, and a fixed sleep
+      # is either too short on a loaded machine or wasted time on a fast one.
+      # Running the poll before gateway_node is what makes it mean anything.
+      # 'ros2 service list' also reports a name that only a client has opened,
+      # and the gateway opens clients for exactly these services, so the same
+      # check made after the gateway is up would pass with nothing serving it.
+      for _ in \$(seq 1 50); do
+        if ros2 service list 2>/dev/null | grep -q '/fault_manager/report_fault'; then
+          break
+        fi
+        sleep 0.2
+      done
+      if ! ros2 service list 2>/dev/null | grep -q '/fault_manager/report_fault'; then
+        echo 'ERROR: fault_manager_node did not advertise /fault_manager/report_fault within 10s.' >&2
+        echo 'Last lines of /var/lib/ros2_medkit/fault_manager.log:' >&2
+        tail -n 20 /var/lib/ros2_medkit/fault_manager.log >&2 || true
+        exit 1
+      fi
+      echo '$FM_READY_MARKER'
       PLUGIN_PATH=\$(find /root/ws/install -name 'libros2_medkit_opcua_plugin.so' | head -1)
       ros2 run ros2_medkit_gateway gateway_node \
         --ros-args --params-file /config/gateway_params.yaml \
@@ -64,6 +108,32 @@ docker run -d --name gateway --network plc-demo -p 8080:8080 \
         -p discovery.manifest_path:=/config/manifest.yaml \
         -p discovery.manifest_strict_validation:=false"
 
+echo "Fault manager starting..."
+fm_ready=0
+for _ in $(seq 1 60); do
+    # Substring match rather than a pipe into grep -q: grep -q closes the pipe
+    # on the first hit, and under `set -o pipefail` the SIGPIPE'd `docker logs`
+    # would turn a found marker into a failed test.
+    if [[ "$(docker logs gateway 2>&1)" == *"$FM_READY_MARKER"* ]]; then
+        fm_ready=1
+        echo "  Fault manager ready (log: /var/lib/ros2_medkit/fault_manager.log)"
+        break
+    fi
+    # The container exits when the wait above timed out. Stop polling for a
+    # marker that can no longer arrive and report it below.
+    if [ -z "$(docker ps -q --filter 'name=^gateway$')" ]; then
+        break
+    fi
+    sleep 1
+done
+if [ "$fm_ready" -eq 0 ]; then
+    echo "ERROR: the fault manager never advertised /fault_manager/report_fault." >&2
+    echo "The demo needs it: without it the alarms the PLC raises are dropped and" >&2
+    echo "/api/v1/faults answers 503. Gateway container log:" >&2
+    docker logs gateway 2>&1 | tail -20 >&2
+    exit 1
+fi
+
 echo "Gateway starting..."
 
 for _ in $(seq 1 30); do
@@ -76,6 +146,8 @@ for _ in $(seq 1 30); do
         echo "Stop:  bash scripts/stop.sh"
         echo "Tests: bash scripts/run_integration_tests.sh"
         echo "State: volume '$STATE_VOLUME' (kept across stop/start)"
+        echo "Faults: fault_manager_node runs in the gateway container"
+        echo "        (log: docker exec gateway cat /var/lib/ros2_medkit/fault_manager.log)"
         exit 0
     fi
     sleep 2