Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 31 additions & 5 deletions docs/tutorials/snapshots.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,32 @@ configuration always overrides the zero-config fallback when present.

.. note::

By default, snapshots are deleted when a fault is cleared via the
``DELETE /api/v1/faults/{code}`` endpoint or ``~/clear_fault`` service.
Set ``snapshots.retain_on_clear: true`` to keep them across clears.
By default, clearing a fault deletes its **value snapshots**, the per-topic
JSON captures this tutorial configures. Every clear reaches storage the same
way, whatever asked for it: the per-fault
``DELETE /api/v1/apps/{app_id}/faults/{fault_code}`` route (``components``,
``areas`` and ``functions`` carry the same one, though there is no global
``DELETE /api/v1/faults/{code}``), the bulk
``DELETE /api/v1/apps/{app_id}/faults`` and ``DELETE /api/v1/faults``, the
``~/clear_fault`` service, and the correlation cascade, which clears a root
cause's symptoms with no clear addressed to them. The OPC UA plugin calls
that same service when a threshold alarm de-asserts, so a value going back
in range clears the fault the way an operator does; a native AlarmCondition
clears that way only once the operator has acknowledged it (and confirmed
it, unless ``require_confirm_for_clear`` is off), and stays CONFIRMED until
then. A plugin that de-asserts with a PASSED report instead, as the graph
watchdog does, goes through debounce and healing, and nothing on that path
deletes the snapshots: with ``healing_enabled`` the fault heals once its
counter reaches ``healing_threshold``, without it (the default) it stays
CONFIRMED, and its snapshots stay either way. Starting with healing disabled takes the
value snapshots of leftover HEALED rows as it reclassifies them, without a
clear at all. ``snapshots.retain_on_clear: true`` keeps the value snapshots
across all of it, and only those. The rosbag recording is not covered by that
setting. A clear deletes the recording under
``snapshots.rosbag.auto_cleanup``, unless that is off or
``snapshots.rosbag.max_bags_per_fault`` keeps a history. The cap then governs
retention, except at ``0`` (unlimited), where only
``snapshots.rosbag.max_total_storage_mb`` bounds it.

Quick Start
-----------
Expand Down Expand Up @@ -360,7 +383,7 @@ Troubleshooting

**Empty topics object in response**

- The fault may have been cleared (snapshots are deleted on clear unless
- The fault may have been cleared (value snapshots are deleted on clear unless
``snapshots.retain_on_clear`` is enabled)
- No topics were configured for this fault code
- All configured topics timed out or exceeded size limit
Expand All @@ -380,7 +403,10 @@ Rosbag Capture (Time-Window Recording)

In addition to JSON snapshots, you can enable **rosbag capture** for "black box"
style recording. This continuously buffers messages in memory and flushes them
to a bag file when a fault is confirmed.
to a bag file when a fault is confirmed. The recording is deleted when the fault
is cleared, unless ``snapshots.rosbag.auto_cleanup`` is off or
``snapshots.rosbag.max_bags_per_fault`` keeps a history, which takes over the
retention decision.

**Key differences from JSON snapshots:**

Expand Down
2 changes: 1 addition & 1 deletion src/ros2_medkit_fault_manager/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
- **Severity escalation**: Fault severity is updated if a higher severity is reported
- **Persistent storage**: SQLite backend ensures faults survive node restarts
- **Debounce filtering** (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity threshold overrides
- **Snapshot capture**: Captures topic data when faults are confirmed for debugging (snapshots are deleted when fault is cleared)
- **Snapshot capture**: Captures topic data when faults are confirmed for debugging (the value snapshots are deleted when the fault is cleared, unless `snapshots.retain_on_clear` is set)
- **Near-miss series**: Appends one entry per FAILED report that moved the debounce counter without confirming, bounded per fault code and retained when the fault is cleared
- **Freeze-frame retention**: One compact JSON freeze-frame per fault code, retained across `clear_fault` (see below)
- **Fault correlation** (optional): Root cause analysis with symptom muting and auto-clear
Expand Down
2 changes: 2 additions & 0 deletions src/ros2_medkit_fault_manager/config/snapshots.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,8 @@ rosbag:
# Auto-cleanup bag files when fault is cleared (default: true)
# When true, bag files are deleted when ClearFault is called
# When false, bag files persist until manually deleted or storage limit hit
# No effect once max_bags_per_fault is anything other than 1: a configured
# history governs retention instead, and an acknowledgement leaves it alone
auto_cleanup: true

# =============================================================================
Expand Down
2 changes: 1 addition & 1 deletion src/ros2_medkit_gateway/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1267,7 +1267,7 @@ Rosbag capture is configured via FaultManager parameters. See `config/snapshots.
| `snapshots.rosbag.exclude_sensor_topics` | bool | `true` | Auto-exclude image/points/depth/compressed in broad modes (`include_topics` re-adds) |
| `snapshots.rosbag.qos_match` | bool | `true` | Match each topic's publisher QoS for faithful capture |
| `snapshots.rosbag.format` | string | `"mcap"` | Bag format: `"mcap"` (default; opens directly in Foxglove and Lichtblick) or `"sqlite3"`. Neither is privileged - an unavailable plugin falls back automatically to the other one |
| `snapshots.rosbag.auto_cleanup` | bool | `true` | Delete bag when fault is cleared |
| `snapshots.rosbag.auto_cleanup` | bool | `true` | Delete a fault's bag when the fault is cleared. No effect once `snapshots.rosbag.max_bags_per_fault` is anything other than `1`, where the cap governs retention instead |
| `snapshots.rosbag.max_buffer_mb` | int | `256` | Ring-buffer RAM cap (oldest messages drop past it) |
| `snapshots.rosbag.max_bag_size_mb` | int | `50` | Max size per bag file |
| `snapshots.rosbag.max_total_storage_mb` | int | `500` | Total storage limit |
Expand Down
9 changes: 8 additions & 1 deletion src/ros2_medkit_plugins/ros2_medkit_opcua/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -875,7 +875,14 @@ GET /api/v1/apps/tank_process/faults
-> [{fault_code: "PLC_HIGH_TEMP", severity: "ERROR", ...}]
```

When the value returns below threshold, the fault is automatically cleared.
When the value returns below threshold, the plugin calls the fault manager's
`~/clear_fault` service (`/fault_manager/clear_fault`) for that fault code. That
is the same service an operator's
`DELETE /api/v1/apps/{app_id}/faults/{fault_code}` ends up calling, so a device
de-assert is a clear like any other. It drops the fault's value snapshots unless
`snapshots.retain_on_clear` is set, and its rosbag recording unless
`snapshots.rosbag.auto_cleanup` is off or `snapshots.rosbag.max_bags_per_fault`
keeps a history.

## Key Design Decisions

Expand Down
Loading