Skip to content

graph_watchdog: value detectors with a window and an enable condition #669

Description

@mfaferek93

Thread: https://discourse.ros.org/t/why-we-did-not-extend-diagnostics/57859

Problem

Five cases from one localization filter with every /diagnostics status at OK: GPS gate rejecting every fix, position drifting backwards at velocity 1.0, NIS at 0.03 so the outlier gate cannot fire, gyro yaw sign inverted against the wheel encoders for months, a gate whose largest statistic stayed 39 times below its threshold for a whole run. No FAILED event, so the fault manager never sees them. Each needs a number checked over a window, plus a rule for when the check is active.

Today thresholds are per sample and only for plugin data points (FaultTriggerRule in the gateway, ros2_medkit_fault_detection in the plugins). Nothing reads topics or /diagnostics key-values. No window, no enable condition.

The watchdog Detector interface (configure(json), tick, gateway subscriptions, REGISTER_DETECTOR) is the place for it.

Change

value detector, one rule per fault code:

  • source: topic field (/odom.twist.twist.angular.z), key-value (diag:imu_filter/rejected_fixes), or an expression over two sources (sign(a) != sign(b))
  • check: threshold, band, or expression result
  • enable: expression, a sample counts only while true (abs(a) > 0.08 and abs(b) > 0.08)
  • window: time (fails for seconds), counter (up step_up on fail, down step_down on pass, confirm at threshold, floor 0), ratio (more than percent failed of the last n enabled samples)
  • fault: code, severity, description, owning entity
  • the mirrored condition reports PASSED, and on FAILED the last n samples of each source go to the freeze frame

monitor_headroom detector: for a rule with a threshold, max of the statistic over the last n enabled samples divided by the threshold. Below min_ratio for n samples raises MONITOR_CANNOT_FIRE on the same entity. This checks the monitor, not the sensor.

Files: src/detectors/value_detector.cpp, src/detectors/monitor_headroom_detector.cpp, keys in detector_config_keys.hpp, field extraction through the introspection the data routes already use, README with the five rules as examples.

Done when

  • A bag with the gyro sign inverted raises YAW_SIGN_DISAGREEMENT with ratio (n=200, percent=80). The correct sign does not (about 6 percent). time at 3 s fires on neither.
  • A rule with enable counts nothing during a GPS blackout section of a bag.
  • monitor_headroom on the GPS gate rule raises MONITOR_CANNOT_FIRE when the max statistic stays under 5 percent of the threshold.
  • Unit tests per window shape, one integration test through the fault manager and REST.

Out of scope: learned thresholds, spike-and-replay routines (SOVD operation, separate issue), time debounce in the fault manager (auto_confirm_after_sec exists).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions