From 0d3ddfd4da8d010ca1c53f477969f3730d938f51 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Tue, 8 Sep 2026 21:44:38 +0200 Subject: [PATCH 1/3] docs: qualify what clearing a fault deletes The snapshots tutorial said a clear deletes "snapshots" and pointed at a DELETE /api/v1/faults/{code} endpoint that is not registered. Two things were wrong. retain_on_clear scopes to the value snapshots only, and the rosbag recording is governed by rosbag.auto_cleanup, with rosbag.max_bags_per_fault taking over the retention decision once a history is configured. The per-fault DELETE route is entity-scoped. Name the service a plugin calls when its device de-asserts an alarm too, so a device going quiet reads as the clear it is, and qualify the same rule where it was repeated in the fault manager, gateway and OPC UA READMEs and in the shipped snapshots.yaml. --- docs/tutorials/snapshots.rst | 23 +++++++++++++++---- src/ros2_medkit_fault_manager/README.md | 2 +- .../config/snapshots.yaml | 2 ++ src/ros2_medkit_gateway/README.md | 2 +- .../ros2_medkit_opcua/README.md | 8 ++++++- 5 files changed, 29 insertions(+), 8 deletions(-) diff --git a/docs/tutorials/snapshots.rst b/docs/tutorials/snapshots.rst index f17d55d7e..6a9da3d9b 100644 --- a/docs/tutorials/snapshots.rst +++ b/docs/tutorials/snapshots.rst @@ -31,9 +31,19 @@ configuration always overrides the zero-config fallback when present. .. note:: - By default, snapshots are deleted when a fault is cleared via the - ``DELETE /api/v1/faults/{code}`` endpoint or ``~/clear_fault`` service. - Set ``snapshots.retain_on_clear: true`` to keep them across clears. + By default, clearing a fault deletes its **value snapshots**, the per-topic + JSON captures this tutorial configures. A clear reaches the fault manager + either from the entity-scoped DELETE route, for example + ``DELETE /api/v1/apps/{app_id}/faults/{fault_code}`` (``components``, + ``areas`` and ``functions`` carry the same route, and there is no global + ``DELETE /api/v1/faults/{code}``), or from the ``~/clear_fault`` service. A + plugin whose device de-asserts an alarm calls that same service, so a device + going quiet clears the fault the way an operator does. + ``snapshots.retain_on_clear: true`` keeps the value snapshots across a clear, + and only those. The rosbag recording follows + ``snapshots.rosbag.auto_cleanup``, which deletes it on clear unless + ``snapshots.rosbag.max_bags_per_fault`` keeps a history, in which case that + cap governs the recording's retention instead. Quick Start ----------- @@ -360,7 +370,7 @@ Troubleshooting **Empty topics object in response** -- The fault may have been cleared (snapshots are deleted on clear unless +- The fault may have been cleared (value snapshots are deleted on clear unless ``snapshots.retain_on_clear`` is enabled) - No topics were configured for this fault code - All configured topics timed out or exceeded size limit @@ -380,7 +390,10 @@ Rosbag Capture (Time-Window Recording) In addition to JSON snapshots, you can enable **rosbag capture** for "black box" style recording. This continuously buffers messages in memory and flushes them -to a bag file when a fault is confirmed. +to a bag file when a fault is confirmed. The recording is deleted when the fault +is cleared, unless ``snapshots.rosbag.auto_cleanup`` is off or +``snapshots.rosbag.max_bags_per_fault`` keeps a history, which takes over the +retention decision. **Key differences from JSON snapshots:** diff --git a/src/ros2_medkit_fault_manager/README.md b/src/ros2_medkit_fault_manager/README.md index c12e54c8a..977f5a8c4 100644 --- a/src/ros2_medkit_fault_manager/README.md +++ b/src/ros2_medkit_fault_manager/README.md @@ -52,7 +52,7 @@ ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \ - **Severity escalation**: Fault severity is updated if a higher severity is reported - **Persistent storage**: SQLite backend ensures faults survive node restarts - **Debounce filtering** (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity threshold overrides -- **Snapshot capture**: Captures topic data when faults are confirmed for debugging (snapshots are deleted when fault is cleared) +- **Snapshot capture**: Captures topic data when faults are confirmed for debugging (the value snapshots are deleted when the fault is cleared, unless `snapshots.retain_on_clear` is set) - **Near-miss series**: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared - **Freeze-frame retention**: One compact JSON freeze-frame per fault code, retained across `clear_fault` (see below) - **Fault correlation** (optional): Root cause analysis with symptom muting and auto-clear diff --git a/src/ros2_medkit_fault_manager/config/snapshots.yaml b/src/ros2_medkit_fault_manager/config/snapshots.yaml index 6e7179f20..a4efadfed 100644 --- a/src/ros2_medkit_fault_manager/config/snapshots.yaml +++ b/src/ros2_medkit_fault_manager/config/snapshots.yaml @@ -192,6 +192,8 @@ rosbag: # Auto-cleanup bag files when fault is cleared (default: true) # When true, bag files are deleted when ClearFault is called # When false, bag files persist until manually deleted or storage limit hit + # No effect once max_bags_per_fault is anything other than 1: a configured + # history governs retention instead, and an acknowledgement leaves it alone auto_cleanup: true # ============================================================================= diff --git a/src/ros2_medkit_gateway/README.md b/src/ros2_medkit_gateway/README.md index 0cff2b0d3..5c0f55990 100644 --- a/src/ros2_medkit_gateway/README.md +++ b/src/ros2_medkit_gateway/README.md @@ -1267,7 +1267,7 @@ Rosbag capture is configured via FaultManager parameters. See `config/snapshots. | `snapshots.rosbag.exclude_sensor_topics` | bool | `true` | Auto-exclude image/points/depth/compressed in broad modes (`include_topics` re-adds) | | `snapshots.rosbag.qos_match` | bool | `true` | Match each topic's publisher QoS for faithful capture | | `snapshots.rosbag.format` | string | `"mcap"` | Bag format: `"mcap"` (default; opens directly in Foxglove and Lichtblick) or `"sqlite3"`. Neither is privileged - an unavailable plugin falls back automatically to the other one | -| `snapshots.rosbag.auto_cleanup` | bool | `true` | Delete bag when fault is cleared | +| `snapshots.rosbag.auto_cleanup` | bool | `true` | Delete a fault's bag when the fault is cleared. No effect once `snapshots.rosbag.max_bags_per_fault` is anything other than `1`, where the cap governs retention instead | | `snapshots.rosbag.max_buffer_mb` | int | `256` | Ring-buffer RAM cap (oldest messages drop past it) | | `snapshots.rosbag.max_bag_size_mb` | int | `50` | Max size per bag file | | `snapshots.rosbag.max_total_storage_mb` | int | `500` | Total storage limit | diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md index 0381ef765..e0a6a3ce1 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md @@ -875,7 +875,13 @@ GET /api/v1/apps/tank_process/faults -> [{fault_code: "PLC_HIGH_TEMP", severity: "ERROR", ...}] ``` -When the value returns below threshold, the fault is automatically cleared. +When the value returns below threshold, the plugin calls the fault manager's +`~/clear_fault` service (`/fault_manager/clear_fault`) for that fault code. That +is the same service an operator's +`DELETE /api/v1/apps/{app_id}/faults/{fault_code}` ends up calling, so a device +de-assert is a clear like any other. It drops the fault's value snapshots unless +`snapshots.retain_on_clear` is set, and its rosbag recording unless +`snapshots.rosbag.max_bags_per_fault` keeps a history. ## Key Design Decisions From 49ff78621ebf7c0192a4c380e7b4b78552f45993 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Tue, 8 Sep 2026 22:05:33 +0200 Subject: [PATCH 2/3] docs: correct the two rosbag gates and stop implying a closed list of clears on_fault_cleared returns on either gate, so a de-assert leaves the recording alone when auto_cleanup is off as well as when max_bags_per_fault keeps a history. The OPC UA README named only the second one. The snapshots note read as an exhaustive list of the ways a fault gets cleared, which the "there is no global DELETE /api/v1/faults/{code}" sentence reinforced. The bulk DELETE routes, the correlation cascade clearing a root cause's symptoms, and the startup reclassification of HEALED rows all reach the same storage path, so name the routes as examples and keep the one true absence. Also bound the max_bags_per_fault claim: at 0 the cap is unlimited and only max_total_storage_mb holds the recordings back. --- docs/tutorials/snapshots.rst | 30 +++++++++++-------- .../ros2_medkit_opcua/README.md | 3 +- 2 files changed, 20 insertions(+), 13 deletions(-) diff --git a/docs/tutorials/snapshots.rst b/docs/tutorials/snapshots.rst index 6a9da3d9b..26bb6f73b 100644 --- a/docs/tutorials/snapshots.rst +++ b/docs/tutorials/snapshots.rst @@ -32,18 +32,24 @@ configuration always overrides the zero-config fallback when present. .. note:: By default, clearing a fault deletes its **value snapshots**, the per-topic - JSON captures this tutorial configures. A clear reaches the fault manager - either from the entity-scoped DELETE route, for example - ``DELETE /api/v1/apps/{app_id}/faults/{fault_code}`` (``components``, - ``areas`` and ``functions`` carry the same route, and there is no global - ``DELETE /api/v1/faults/{code}``), or from the ``~/clear_fault`` service. A - plugin whose device de-asserts an alarm calls that same service, so a device - going quiet clears the fault the way an operator does. - ``snapshots.retain_on_clear: true`` keeps the value snapshots across a clear, - and only those. The rosbag recording follows - ``snapshots.rosbag.auto_cleanup``, which deletes it on clear unless - ``snapshots.rosbag.max_bags_per_fault`` keeps a history, in which case that - cap governs the recording's retention instead. + JSON captures this tutorial configures. Every clear reaches storage the same + way, whatever asked for it: the per-fault + ``DELETE /api/v1/apps/{app_id}/faults/{fault_code}`` route (``components``, + ``areas`` and ``functions`` carry the same one, though there is no global + ``DELETE /api/v1/faults/{code}``), the bulk + ``DELETE /api/v1/apps/{app_id}/faults`` and ``DELETE /api/v1/faults``, the + ``~/clear_fault`` service, and the correlation cascade, which clears a root + cause's symptoms with no clear addressed to them. A plugin calls that same + service when its device de-asserts an alarm, so a device going quiet clears + the fault the way an operator does. Starting with healing disabled takes the + value snapshots of leftover HEALED rows as it reclassifies them, without a + clear at all. ``snapshots.retain_on_clear: true`` keeps the value snapshots + across all of it, and only those. The rosbag recording is not covered by that + setting. A clear deletes the recording under + ``snapshots.rosbag.auto_cleanup``, unless that is off or + ``snapshots.rosbag.max_bags_per_fault`` keeps a history. The cap then governs + retention, except at ``0`` (unlimited), where only + ``snapshots.rosbag.max_total_storage_mb`` bounds it. Quick Start ----------- diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md index e0a6a3ce1..bf262d56e 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md @@ -881,7 +881,8 @@ is the same service an operator's `DELETE /api/v1/apps/{app_id}/faults/{fault_code}` ends up calling, so a device de-assert is a clear like any other. It drops the fault's value snapshots unless `snapshots.retain_on_clear` is set, and its rosbag recording unless -`snapshots.rosbag.max_bags_per_fault` keeps a history. +`snapshots.rosbag.auto_cleanup` is off or `snapshots.rosbag.max_bags_per_fault` +keeps a history. ## Key Design Decisions From 81b1f28e4b8e55ba6d49470236a01e2fddedb390 Mon Sep 17 00:00:00 2001 From: Bartosz Burda Date: Fri, 11 Sep 2026 11:15:52 +0200 Subject: [PATCH 3/3] docs: name which de-asserts clear a fault and which heal it The OPC UA plugin calls clear_fault when a threshold alarm de-asserts. A native AlarmCondition stays CONFIRMED until the operator acknowledges it and, unless require_confirm_for_clear is off, confirms it. A plugin that de-asserts with a PASSED report, as the graph watchdog does, goes through debounce and healing, where nothing deletes snapshots: with healing_enabled the fault heals once its counter reaches healing_threshold, and with the default it stays CONFIRMED. --- docs/tutorials/snapshots.rst | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/tutorials/snapshots.rst b/docs/tutorials/snapshots.rst index 26bb6f73b..1070aa373 100644 --- a/docs/tutorials/snapshots.rst +++ b/docs/tutorials/snapshots.rst @@ -39,9 +39,16 @@ configuration always overrides the zero-config fallback when present. ``DELETE /api/v1/faults/{code}``), the bulk ``DELETE /api/v1/apps/{app_id}/faults`` and ``DELETE /api/v1/faults``, the ``~/clear_fault`` service, and the correlation cascade, which clears a root - cause's symptoms with no clear addressed to them. A plugin calls that same - service when its device de-asserts an alarm, so a device going quiet clears - the fault the way an operator does. Starting with healing disabled takes the + cause's symptoms with no clear addressed to them. The OPC UA plugin calls + that same service when a threshold alarm de-asserts, so a value going back + in range clears the fault the way an operator does; a native AlarmCondition + clears that way only once the operator has acknowledged it (and confirmed + it, unless ``require_confirm_for_clear`` is off), and stays CONFIRMED until + then. A plugin that de-asserts with a PASSED report instead, as the graph + watchdog does, goes through debounce and healing, and nothing on that path + deletes the snapshots: with ``healing_enabled`` the fault heals once its + counter reaches ``healing_threshold``, without it (the default) it stays + CONFIRMED, and its snapshots stay either way. Starting with healing disabled takes the value snapshots of leftover HEALED rows as it reclassifies them, without a clear at all. ``snapshots.retain_on_clear: true`` keeps the value snapshots across all of it, and only those. The rosbag recording is not covered by that