Thread: https://discourse.ros.org/t/why-we-did-not-extend-diagnostics/57859
Problem
Five cases from one localization filter with every /diagnostics status at OK: GPS gate rejecting every fix, position drifting backwards at velocity 1.0, NIS at 0.03 so the outlier gate cannot fire, gyro yaw sign inverted against the wheel encoders for months, a gate whose largest statistic stayed 39 times below its threshold for a whole run. No FAILED event, so the fault manager never sees them. Each needs a number checked over a window, plus a rule for when the check is active.
Today thresholds are per sample and only for plugin data points (FaultTriggerRule in the gateway, ros2_medkit_fault_detection in the plugins). Nothing reads topics or /diagnostics key-values. No window, no enable condition.
The watchdog Detector interface (configure(json), tick, gateway subscriptions, REGISTER_DETECTOR) is the place for it.
Change
value detector, one rule per fault code:
source: topic field (/odom.twist.twist.angular.z), key-value (diag:imu_filter/rejected_fixes), or an expression over two sources (sign(a) != sign(b))
check: threshold, band, or expression result
enable: expression, a sample counts only while true (abs(a) > 0.08 and abs(b) > 0.08)
window: time (fails for seconds), counter (up step_up on fail, down step_down on pass, confirm at threshold, floor 0), ratio (more than percent failed of the last n enabled samples)
fault: code, severity, description, owning entity
- the mirrored condition reports PASSED, and on FAILED the last
n samples of each source go to the freeze frame
monitor_headroom detector: for a rule with a threshold, max of the statistic over the last n enabled samples divided by the threshold. Below min_ratio for n samples raises MONITOR_CANNOT_FIRE on the same entity. This checks the monitor, not the sensor.
Files: src/detectors/value_detector.cpp, src/detectors/monitor_headroom_detector.cpp, keys in detector_config_keys.hpp, field extraction through the introspection the data routes already use, README with the five rules as examples.
Done when
- A bag with the gyro sign inverted raises
YAW_SIGN_DISAGREEMENT with ratio (n=200, percent=80). The correct sign does not (about 6 percent). time at 3 s fires on neither.
- A rule with
enable counts nothing during a GPS blackout section of a bag.
monitor_headroom on the GPS gate rule raises MONITOR_CANNOT_FIRE when the max statistic stays under 5 percent of the threshold.
- Unit tests per window shape, one integration test through the fault manager and REST.
Out of scope: learned thresholds, spike-and-replay routines (SOVD operation, separate issue), time debounce in the fault manager (auto_confirm_after_sec exists).
Thread: https://discourse.ros.org/t/why-we-did-not-extend-diagnostics/57859
Problem
Five cases from one localization filter with every
/diagnosticsstatus at OK: GPS gate rejecting every fix, position drifting backwards at velocity 1.0, NIS at 0.03 so the outlier gate cannot fire, gyro yaw sign inverted against the wheel encoders for months, a gate whose largest statistic stayed 39 times below its threshold for a whole run. No FAILED event, so the fault manager never sees them. Each needs a number checked over a window, plus a rule for when the check is active.Today thresholds are per sample and only for plugin data points (
FaultTriggerRulein the gateway,ros2_medkit_fault_detectionin the plugins). Nothing reads topics or/diagnosticskey-values. No window, no enable condition.The watchdog
Detectorinterface (configure(json),tick, gateway subscriptions,REGISTER_DETECTOR) is the place for it.Change
valuedetector, one rule per fault code:source: topic field (/odom.twist.twist.angular.z), key-value (diag:imu_filter/rejected_fixes), or an expression over two sources (sign(a) != sign(b))check: threshold, band, or expression resultenable: expression, a sample counts only while true (abs(a) > 0.08 and abs(b) > 0.08)window:time(fails forseconds),counter(upstep_upon fail, downstep_downon pass, confirm atthreshold, floor 0),ratio(more thanpercentfailed of the lastnenabled samples)fault: code, severity, description, owning entitynsamples of each source go to the freeze framemonitor_headroomdetector: for a rule with a threshold, max of the statistic over the lastnenabled samples divided by the threshold. Belowmin_ratiofornsamples raisesMONITOR_CANNOT_FIREon the same entity. This checks the monitor, not the sensor.Files:
src/detectors/value_detector.cpp,src/detectors/monitor_headroom_detector.cpp, keys indetector_config_keys.hpp, field extraction through the introspection the data routes already use, README with the five rules as examples.Done when
YAW_SIGN_DISAGREEMENTwithratio(n=200, percent=80). The correct sign does not (about 6 percent).timeat 3 s fires on neither.enablecounts nothing during a GPS blackout section of a bag.monitor_headroomon the GPS gate rule raisesMONITOR_CANNOT_FIREwhen the max statistic stays under 5 percent of the threshold.Out of scope: learned thresholds, spike-and-replay routines (SOVD operation, separate issue), time debounce in the fault manager (
auto_confirm_after_secexists).