diff --git a/docs/tutorials/snapshots.rst b/docs/tutorials/snapshots.rst index f17d55d7e..1070aa373 100644 --- a/docs/tutorials/snapshots.rst +++ b/docs/tutorials/snapshots.rst @@ -31,9 +31,32 @@ configuration always overrides the zero-config fallback when present. .. note:: - By default, snapshots are deleted when a fault is cleared via the - ``DELETE /api/v1/faults/{code}`` endpoint or ``~/clear_fault`` service. - Set ``snapshots.retain_on_clear: true`` to keep them across clears. + By default, clearing a fault deletes its **value snapshots**, the per-topic + JSON captures this tutorial configures. Every clear reaches storage the same + way, whatever asked for it: the per-fault + ``DELETE /api/v1/apps/{app_id}/faults/{fault_code}`` route (``components``, + ``areas`` and ``functions`` carry the same one, though there is no global + ``DELETE /api/v1/faults/{code}``), the bulk + ``DELETE /api/v1/apps/{app_id}/faults`` and ``DELETE /api/v1/faults``, the + ``~/clear_fault`` service, and the correlation cascade, which clears a root + cause's symptoms with no clear addressed to them. The OPC UA plugin calls + that same service when a threshold alarm de-asserts, so a value going back + in range clears the fault the way an operator does; a native AlarmCondition + clears that way only once the operator has acknowledged it (and confirmed + it, unless ``require_confirm_for_clear`` is off), and stays CONFIRMED until + then. A plugin that de-asserts with a PASSED report instead, as the graph + watchdog does, goes through debounce and healing, and nothing on that path + deletes the snapshots: with ``healing_enabled`` the fault heals once its + counter reaches ``healing_threshold``, without it (the default) it stays + CONFIRMED, and its snapshots stay either way. Starting with healing disabled takes the + value snapshots of leftover HEALED rows as it reclassifies them, without a + clear at all. ``snapshots.retain_on_clear: true`` keeps the value snapshots + across all of it, and only those. The rosbag recording is not covered by that + setting. A clear deletes the recording under + ``snapshots.rosbag.auto_cleanup``, unless that is off or + ``snapshots.rosbag.max_bags_per_fault`` keeps a history. The cap then governs + retention, except at ``0`` (unlimited), where only + ``snapshots.rosbag.max_total_storage_mb`` bounds it. Quick Start ----------- @@ -360,7 +383,7 @@ Troubleshooting **Empty topics object in response** -- The fault may have been cleared (snapshots are deleted on clear unless +- The fault may have been cleared (value snapshots are deleted on clear unless ``snapshots.retain_on_clear`` is enabled) - No topics were configured for this fault code - All configured topics timed out or exceeded size limit @@ -380,7 +403,10 @@ Rosbag Capture (Time-Window Recording) In addition to JSON snapshots, you can enable **rosbag capture** for "black box" style recording. This continuously buffers messages in memory and flushes them -to a bag file when a fault is confirmed. +to a bag file when a fault is confirmed. The recording is deleted when the fault +is cleared, unless ``snapshots.rosbag.auto_cleanup`` is off or +``snapshots.rosbag.max_bags_per_fault`` keeps a history, which takes over the +retention decision. **Key differences from JSON snapshots:** diff --git a/src/ros2_medkit_fault_manager/README.md b/src/ros2_medkit_fault_manager/README.md index c12e54c8a..977f5a8c4 100644 --- a/src/ros2_medkit_fault_manager/README.md +++ b/src/ros2_medkit_fault_manager/README.md @@ -52,7 +52,7 @@ ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \ - **Severity escalation**: Fault severity is updated if a higher severity is reported - **Persistent storage**: SQLite backend ensures faults survive node restarts - **Debounce filtering** (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity threshold overrides -- **Snapshot capture**: Captures topic data when faults are confirmed for debugging (snapshots are deleted when fault is cleared) +- **Snapshot capture**: Captures topic data when faults are confirmed for debugging (the value snapshots are deleted when the fault is cleared, unless `snapshots.retain_on_clear` is set) - **Near-miss series**: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared - **Freeze-frame retention**: One compact JSON freeze-frame per fault code, retained across `clear_fault` (see below) - **Fault correlation** (optional): Root cause analysis with symptom muting and auto-clear diff --git a/src/ros2_medkit_fault_manager/config/snapshots.yaml b/src/ros2_medkit_fault_manager/config/snapshots.yaml index 6e7179f20..a4efadfed 100644 --- a/src/ros2_medkit_fault_manager/config/snapshots.yaml +++ b/src/ros2_medkit_fault_manager/config/snapshots.yaml @@ -192,6 +192,8 @@ rosbag: # Auto-cleanup bag files when fault is cleared (default: true) # When true, bag files are deleted when ClearFault is called # When false, bag files persist until manually deleted or storage limit hit + # No effect once max_bags_per_fault is anything other than 1: a configured + # history governs retention instead, and an acknowledgement leaves it alone auto_cleanup: true # ============================================================================= diff --git a/src/ros2_medkit_gateway/README.md b/src/ros2_medkit_gateway/README.md index 0cff2b0d3..5c0f55990 100644 --- a/src/ros2_medkit_gateway/README.md +++ b/src/ros2_medkit_gateway/README.md @@ -1267,7 +1267,7 @@ Rosbag capture is configured via FaultManager parameters. See `config/snapshots. | `snapshots.rosbag.exclude_sensor_topics` | bool | `true` | Auto-exclude image/points/depth/compressed in broad modes (`include_topics` re-adds) | | `snapshots.rosbag.qos_match` | bool | `true` | Match each topic's publisher QoS for faithful capture | | `snapshots.rosbag.format` | string | `"mcap"` | Bag format: `"mcap"` (default; opens directly in Foxglove and Lichtblick) or `"sqlite3"`. Neither is privileged - an unavailable plugin falls back automatically to the other one | -| `snapshots.rosbag.auto_cleanup` | bool | `true` | Delete bag when fault is cleared | +| `snapshots.rosbag.auto_cleanup` | bool | `true` | Delete a fault's bag when the fault is cleared. No effect once `snapshots.rosbag.max_bags_per_fault` is anything other than `1`, where the cap governs retention instead | | `snapshots.rosbag.max_buffer_mb` | int | `256` | Ring-buffer RAM cap (oldest messages drop past it) | | `snapshots.rosbag.max_bag_size_mb` | int | `50` | Max size per bag file | | `snapshots.rosbag.max_total_storage_mb` | int | `500` | Total storage limit | diff --git a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md index 0381ef765..bf262d56e 100644 --- a/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md +++ b/src/ros2_medkit_plugins/ros2_medkit_opcua/README.md @@ -875,7 +875,14 @@ GET /api/v1/apps/tank_process/faults -> [{fault_code: "PLC_HIGH_TEMP", severity: "ERROR", ...}] ``` -When the value returns below threshold, the fault is automatically cleared. +When the value returns below threshold, the plugin calls the fault manager's +`~/clear_fault` service (`/fault_manager/clear_fault`) for that fault code. That +is the same service an operator's +`DELETE /api/v1/apps/{app_id}/faults/{fault_code}` ends up calling, so a device +de-assert is a clear like any other. It drops the fault's value snapshots unless +`snapshots.retain_on_clear` is set, and its rosbag recording unless +`snapshots.rosbag.auto_cleanup` is off or `snapshots.rosbag.max_bags_per_fault` +keeps a history. ## Key Design Decisions