diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/CHANGELOG.rst b/src/ros2_medkit_plugins/ros2_medkit_opcua/CHANGELOG.rst index 1db5561be..276145535 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/CHANGELOG.rst +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/CHANGELOG.rst @@ -10,6 +10,7 @@ Forthcoming * ``PLC_COMMS_LOST`` is cleared on every successful connect, including the first, so a fault ``fault_manager`` persisted before a restart does not stand against a healthy link. That clear, and the per-entity ``DELETE /{entity}/faults/{code}`` route when it is served by this plugin, set ``skip_correlation_auto_clear``: neither is an operator resolving a root cause, so neither may cascade-clear correlated symptom faults reported by apps in other entities. A clear reported by the device itself still cascades * Fault dispatches buffered while ``fault_manager`` is unreachable no longer lose one-shot events. Only the link-state ``PLC_COMMS_LOST`` clear is re-derivable (the next reconnect sends it again), so it is what the bounded buffer gives up first. Alarm reports, a device alarm's inactive edge and an operator's scoped clear age out oldest-first as before, and at most one clear per fault code is pending at a time * Gateway-side changes that land on this plugin's entities: a freeze-frame captured from a plugin entity now names the path that read the values in ``x-medkit.source`` (``plugin_data_provider`` or ``plugin_x_plc_data_route``, the only provenance an entity frame carries since it has no ROS topic), and the gateway no longer lists its own in-process helper nodes among the discovered apps +* The OpenPLC tank demo (``docker/scripts/start.sh``) now runs a ``fault_manager_node`` beside the gateway, so an alarm the PLC raises becomes a fault the SOVD API serves. Without one ``GET /api/v1/faults`` answered 503 and the per-entity list came back empty, so the demo could show live PLC values and never a fault. Start-up waits for ``/fault_manager/report_fault`` to be advertised and fails with a clear message when it never appears, and ``faults.db`` is written to the same named volume as the entity freeze frames, so a fault and the frame it carries both outlive ``scripts/stop.sh`` * Contributors: @bburda 0.7.0 (2026-08-27) diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md index e3f142d3f..462b6145c 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md @@ -862,12 +862,15 @@ The plugin ships a self-contained OpenPLC tank demo in `docker/` that exercises ```bash cd src/ros2_medkit_plugins/ros2_medkit_opcua/docker -# Start OpenPLC + gateway (builds everything) +# Start OpenPLC + gateway + fault manager (builds everything) bash scripts/start.sh # Manual testing curl -s http://localhost:8080/api/v1/apps/tank_process/x-plc-data | jq . +# The faults the PLC's alarms raise +curl -s http://localhost:8080/api/v1/apps/tank_process/faults | jq . + # Automated tests (16 assertions) bash scripts/run_integration_tests.sh @@ -875,9 +878,22 @@ bash scripts/run_integration_tests.sh bash scripts/stop.sh ``` +`start.sh` runs a `fault_manager_node` inside the gateway container, on the same +ROS domain, started before `gateway_node` and waited on until +`/fault_manager/report_fault` is advertised. That is the other half of the +"Alarm-to-Fault Bridge" below: the plugin detects the alarm, the fault manager +is what holds the fault. Without one `GET /api/v1/faults` answers 503 and the +per-entity list comes back empty, so the demo can show live PLC values but +never a fault. Its log is inside the container: + +```bash +docker exec gateway cat /var/lib/ros2_medkit/fault_manager.log +``` + `start.sh` mounts the named volume `ros2-medkit-opcua-state` at `/var/lib/ros2_medkit`, so the gateway's state survives `stop.sh` and the next -`start.sh`: the entity freeze frames, `faults.db` and the rosbags. That is what +`start.sh`: the entity freeze frames, `faults.db`, and the rosbags when +black-box capture is enabled (it is opt-in and off by default). That is what lets a fault raised before the stop still serve the values frozen when it confirmed, with its original `captured_at` and no `x-medkit.capture_origin`, rather than a fresh read of the PLC as it is after the restart. `stop.sh` diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/gateway_params.yaml b/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/gateway_params.yaml index f03bc668a..13f7cfa17 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/gateway_params.yaml +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/gateway_params.yaml @@ -17,9 +17,16 @@ ros2_medkit_gateway: # confirmed instead of re-reading the PLC as it is now. scripts/start.sh # mounts the named volume "ros2-medkit-opcua-state" here, which is what # carries the frames across scripts/stop.sh too: that script removes the - # container, so the writable layer would not survive it. The test scripts - # start their own containers without a volume and keep the frames inside - # the container, which is all a single-run suite needs. + # container, so the writable layer would not survive it. The fault manager + # scripts/start.sh runs beside the gateway writes its "faults.db" into the + # same directory, so the fault and the frame it points at cross a restart + # together. A frame without its fault is unreachable, the fault detail + # route is what serves it. The test scripts start their own containers + # without a volume, so the frames stay inside the container and go with it. + # run_integration_tests.sh and test_all.sh run no fault manager, so there is + # no "faults.db" there at all, and the suites that do start one + # (run_alarm_tests.sh, run_discovery_race_test.sh) leave it in the container + # too. That is all a single-run suite needs. entity_freeze_frame: storage: path: "/var/lib/ros2_medkit/entity_freeze_frames.db" diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/start.sh b/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/start.sh index bac6fc6ca..80adc53f1 100755 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/start.sh +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/start.sh @@ -3,6 +3,9 @@ # Usage: from the ros2_medkit repo root, run # bash src/ros2_medkit_plugins/ros2_medkit_opcua/docker/scripts/start.sh # +# The gateway container also runs a fault_manager_node, so an alarm the PLC +# raises becomes a fault the SOVD API serves instead of a 503. +# # The gateway's state (entity freeze frames, faults.db, rosbags) is kept on the # named volume below, so it survives stop.sh and a later start.sh. Purge it with # docker volume rm ros2-medkit-opcua-state @@ -10,6 +13,11 @@ set -eo pipefail STATE_VOLUME="${OPCUA_DEMO_STATE_VOLUME:-ros2-medkit-opcua-state}" +# Printed by the gateway container once fault_manager_node has advertised its +# services, and grepped for below. The container is where the wait happens, this +# is how the host learns the outcome. +FM_READY_MARKER="fault_manager ready: /fault_manager/report_fault" + SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" DOCKER_DIR="$(dirname "$SCRIPT_DIR")" PLUGIN_DIR="$(dirname "$DOCKER_DIR")" @@ -56,6 +64,42 @@ docker run -d --name gateway --network plc-demo -p 8080:8080 \ mkdir -p /var/lib/ros2_medkit/rosbags /config echo 'manifest_version: \"1.0\"' > /config/manifest.yaml source /opt/ros/jazzy/setup.bash && source /root/ws/install/setup.bash + # A fault manager runs beside the gateway, in the same container and on + # the same ROS domain. Without one the alarms the OPC-UA plugin detects + # have nowhere to go: /api/v1/faults answers 503 and the demo can show + # live PLC values but never a fault. It is started before gateway_node so + # its services are advertised before the plugin calls + # /fault_manager/report_fault. + # + # database_path is passed explicitly rather than left to the default: it + # has to land on /var/lib/ros2_medkit, the named volume, so faults.db + # outlives the container the way the entity freeze frames already do. + # Rosbag capture is opt-in and stays off here. storage_path names the + # same volume, so a recording lands there whenever it is switched on. + ros2 run ros2_medkit_fault_manager fault_manager_node --ros-args \ + -p database_path:=/var/lib/ros2_medkit/faults.db \ + -p snapshots.rosbag.storage_path:=/var/lib/ros2_medkit/rosbags \ + > /var/lib/ros2_medkit/fault_manager.log 2>&1 & + # Poll for the service instead of sleeping a fixed time: 'ros2 service + # list' is the cheapest ROS-native availability signal, and a fixed sleep + # is either too short on a loaded machine or wasted time on a fast one. + # Running the poll before gateway_node is what makes it mean anything. + # 'ros2 service list' also reports a name that only a client has opened, + # and the gateway opens clients for exactly these services, so the same + # check made after the gateway is up would pass with nothing serving it. + for _ in \$(seq 1 50); do + if ros2 service list 2>/dev/null | grep -q '/fault_manager/report_fault'; then + break + fi + sleep 0.2 + done + if ! ros2 service list 2>/dev/null | grep -q '/fault_manager/report_fault'; then + echo 'ERROR: fault_manager_node did not advertise /fault_manager/report_fault within 10s.' >&2 + echo 'Last lines of /var/lib/ros2_medkit/fault_manager.log:' >&2 + tail -n 20 /var/lib/ros2_medkit/fault_manager.log >&2 || true + exit 1 + fi + echo '$FM_READY_MARKER' PLUGIN_PATH=\$(find /root/ws/install -name 'libros2_medkit_opcua_plugin.so' | head -1) ros2 run ros2_medkit_gateway gateway_node \ --ros-args --params-file /config/gateway_params.yaml \ @@ -64,6 +108,32 @@ docker run -d --name gateway --network plc-demo -p 8080:8080 \ -p discovery.manifest_path:=/config/manifest.yaml \ -p discovery.manifest_strict_validation:=false" +echo "Fault manager starting..." +fm_ready=0 +for _ in $(seq 1 60); do + # Substring match rather than a pipe into grep -q: grep -q closes the pipe + # on the first hit, and under `set -o pipefail` the SIGPIPE'd `docker logs` + # would turn a found marker into a failed test. + if [[ "$(docker logs gateway 2>&1)" == *"$FM_READY_MARKER"* ]]; then + fm_ready=1 + echo " Fault manager ready (log: /var/lib/ros2_medkit/fault_manager.log)" + break + fi + # The container exits when the wait above timed out. Stop polling for a + # marker that can no longer arrive and report it below. + if [ -z "$(docker ps -q --filter 'name=^gateway$')" ]; then + break + fi + sleep 1 +done +if [ "$fm_ready" -eq 0 ]; then + echo "ERROR: the fault manager never advertised /fault_manager/report_fault." >&2 + echo "The demo needs it: without it the alarms the PLC raises are dropped and" >&2 + echo "/api/v1/faults answers 503. Gateway container log:" >&2 + docker logs gateway 2>&1 | tail -20 >&2 + exit 1 +fi + echo "Gateway starting..." for _ in $(seq 1 30); do @@ -76,6 +146,8 @@ for _ in $(seq 1 30); do echo "Stop: bash scripts/stop.sh" echo "Tests: bash scripts/run_integration_tests.sh" echo "State: volume '$STATE_VOLUME' (kept across stop/start)" + echo "Faults: fault_manager_node runs in the gateway container" + echo " (log: docker exec gateway cat /var/lib/ros2_medkit/fault_manager.log)" exit 0 fi sleep 2