feat(gateway): keep a plugin entity's freeze frame across a gateway restart - #680
Draft
bburda wants to merge 6 commits into
Draft
feat(gateway): keep a plugin entity's freeze frame across a gateway restart#680bburda wants to merge 6 commits into
bburda wants to merge 6 commits into
Conversation
The frames captured for plugin-backed entities lived in process memory, so a restart threw them away and the startup catch-up re-read the plant as it is now. What came back was the current values under the original fault, stamped with the restart and marked capture_origin: startup. The values at fault time, which are the whole point of a freeze-frame, were gone, and a consumer had no way to tell that the numbers under a fault from last night were read this morning. EntityFreezeFrameCapture now takes an EntityFreezeFrameStore. It writes a fault's frames through on every capture, under the same lock as the map, and loads them back in its constructor, so a reloaded frame is served exactly as it was captured: its original captured_at, its own capture_origin (absent for a confirm-edge frame, startup for one the catch-up took), and the connected and source_timestamp provenance it carried. The startup catch-up then skips any fault that already has a frame and re-reads only the ones with none, so a fault that confirmed while the gateway was down still gets its startup frame. Two backends sit behind the interface, SQLite for the gateway and an in-memory one for tests. entity_freeze_frame.storage.path names the file. Empty puts it next to triggers.storage.path, and with neither set the frames stay in memory, exactly as they were before. A store that cannot be opened or written is reported and the capture keeps working from memory. Bounds. The retained-frame bound of 256 faults counts reloaded and freshly captured frames together, so a reloaded frame no longer spends catch-up budget it does not need, and an evicted fault loses its rows as well as its map entry so the bound means something across a restart. At startup, frames whose fault the fault manager no longer holds in any status are dropped. A fault reported as cleared keeps its frame, and when the fault manager cannot be asked at all nothing is dropped.
…e the gateway was down A persisted frame is reloaded at start and the catch-up skips any fault that already has one. That is right while the fault stands, and wrong the moment it cleared and confirmed again in the meantime. The stored frame then holds the previous incident's values, and the gateway served them under the new occurrence with no marker at all, so nothing on the wire said the numbers came from a different event than the fault being read. The standing-fault list now carries each fault's first_occurred, which the fault manager resets only when a CLEARED fault reactivates. A reloaded frame whose captured_at predates it belongs to an occurrence that has ended, so the row is dropped before the catch-up decides what is already framed. The fault is then re-read now and the result marked capture_origin: startup, exactly what an unframed standing fault has always got. last_occurred cannot serve this. It moves on every FAILED report, so on a fault that keeps failing it is always newer than the row and every standing fault would be re-read at every restart, which is the behaviour persisting the frame is here to replace. A value the reply does not carry reads as "cannot tell" and keeps the stored frame. Also announces the parameter in the changelog, and folds the one-line "exactly one frame per fault" paragraph into the persistence paragraph it belongs to.
The docker demo set neither entity_freeze_frame.storage.path nor triggers.storage.path, so the gateway reported that the frames are not persisted and kept them in process memory. The start and test scripts already create /var/lib/ros2_medkit for the fault manager, so the store goes there and a restart of the demo gateway serves the values frozen when the alarm confirmed instead of re-reading the PLC as it is now.
…say when it goes The re-occurrence fix erased the stale row from memory and from the store and only then let the catch-up try to re-read the entity. When the entity could not answer, which is a restart while the PLC link is down, the fault ended with no frame in memory, no row on disk and not one line about it. The route path returns nothing without a message, the failed capture was neither counted nor logged, and the drop's own INFO claimed the catch-up re-reads those entities as a fact it never checked. The order is now read, then decide. Stale codes are identified without touching anything, the catch-up re-reads them like any unframed fault, and a successful read replaces the frame and its row in one write, marked startup. Only a read that yields nothing drops the row, and then it warns with the fault code, the entity and why, because the operator is losing evidence and the log is the only place they can learn it. The tail INFO counts what happened, how many were re-read and how many went with no replacement. Two things that follow from the new order. A fault with no reporting sources was stale to the comparison and invisible to the catch-up, so its row was dropped by one test and never re-read by the other. It is now a re-read that cannot be attempted and takes the same path and the same warning. A stale code also jumps the queued-confirm skip: that skip saves one plugin read per confirm, and here the alternative is leaving a dead occurrence's frame in place on the chance the drain loop succeeds. Also guards the seconds-to-nanoseconds conversion of first_occurred against a double outside int64's range, where the cast is undefined, and documents the two windows the comparison deliberately leaves alone: a fault that re-failed without re-confirming is not in the confirmed list and keeps its frame until it confirms, and a HEALED to FAILED cycle does not reset first_occurred.
… anything Re-reading a stale frame in the same pass as the faults that have none put the retained-frame bound in front of a read that cannot cost anything, and made the occupancy it works from wrong for everyone else. A stale code already holds a slot in frames_, and capture_for_event evicts only when the code is new to the map, so its replacement can neither exceed the bound nor push another frame out. Counting it as absent therefore did two things at once at a full bound. A fault with no frame at all was admitted against the under-count and the FIFO evicted the front of the insertion order, which is the OLDEST live reloaded frame, from memory and from disk, without naming it. And the stale code itself then hit the bound check and was discarded with a warning saying its entity could not be re-read, when the entity had never been asked. The catch-up now runs in two passes. First every stale code is re-read in place, with no bound check and no eviction, replaced on success and dropped with the existing warning on failure, which frees the slot it was holding. Only then is the occupancy read, from what frames_ really holds, and the faults with no frame are admitted against that. Eviction can no longer reach a code that is stale or mid-re-read, and the bound branch for stale codes is gone because it can no longer be reached. The truncation warning now names the fault codes it refused (up to ten) instead of only counting them. A count leaves an operator knowing that some fault details lack their context but not which. Also gives the range guard on the seconds-to-nanoseconds conversion a test. Reverting it left the suite green, so it guarded nothing that was checked: 1e19 seconds and +inf both reach the cast, and both come back as INT64_MIN on this box, which reads as "older than every frame" and would discard every reloaded frame the reply mentions. NaN is refused by the sign test either way.
…t twice Splitting the catch-up into two passes left a code the first pass had dropped looking, to the second, exactly like a fault that never had a frame. It is absent from frames_ for that very reason, so with a slot free the second pass called the plugin again for the same entity: a second blocking read of a link that had just failed to answer, once per unreachable stale entity at every start. When the link came back between the two reads it was worse than wasteful. The fault ended up holding a fresh frame in memory and on disk while the warning had already told the operator the stored frame was discarded and this occurrence had none, the summary still counted it as discarded with no replacement, and the slot the drop had freed went back to the code that lost it instead of to the fault still waiting for one. The first pass now records every code it settles, replaced or dropped alike, and the second skips them. One read per fault per catch-up, a dropped frame stays dropped for that catch-up, and its slot goes to the next fault with no frame, so the counters and the log lines say what actually happened. The branch's own test hid this behind EXPECT_GE on the read count, and only passed at all because in that one shape the other fault reached the second pass first and the bound then turned the stale code away. It now pins the count exactly, alongside the two shapes that show the defect directly. Also corrects a comment that had the failure mode backwards. An out-of-range cast is undefined, and it is the platform's answer that decides what happens: x86-64 gives INT64_MIN, which the non-positive check rejects, so the frame survives by luck, while a saturating target gives a large positive value that passes that check, sits above every captured_at and discards a good frame. The range guard is what makes the two behave the same.
This was referenced Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A plugin entity's freeze frame lived in gateway memory only. A restart replaced it with a re-read of current values under a new
captured_at, in the same array shape.The gateway now keeps entity frames in a SQLite file,
entity_freeze_frames.dbnext to the trigger store: one row per fault code and entity, written on every capture, kept across a clear, dropped at eviction and at startup for fault codes the fault manager no longer holds. The reload runs before the startup catch-up. A reloaded frame is served as captured. The catch-up re-reads only faults with no frame.A reloaded frame older than the fault's
first_occurredbelongs to an earlier occurrence (the fault cleared and confirmed again while the gateway was down). The catch-up re-reads that entity first, in the slot the frame holds. With values, it replaces the frame in memory and on disk, markedstartup. With nothing (plugin unreachable, all values null) it drops the frame and logs one warning naming the fault and the entity. Each entity is asked once per catch-up.first_occurredis the comparator because it resets only on a new occurrence, whilelast_occurredadvances on every failing report.entity_freeze_frame.storage.pathdefaults next totriggers.storage.path. With neither set, frames stay in memory and the gateway logs that. An unopenable store is logged and capture continues in memory. The OPC UA plugin's docker config sets the path under/var/lib/ros2_medkit,start.shmounts a named volume there so state survivesstop.shandstart.sh, and the gateway changelog announces the parameter.Known limits: one frame per fault, the newest. A fault that re-failed but has not re-confirmed keeps its earlier frame until it confirms. A HEALED to FAILED cycle does not reset
first_occurred(healing is off by default).Stacked on #660.
Issue
Type
Testing
Store suite: 14 tests over both backends, reopen and a corrupted row included.
Capture suite: 17 new tests. Write-through, replace on re-confirm, a reloaded frame served as captured, the catch-up skipping reloaded codes, the prune of unknown codes, the bound counting reloaded frames, eviction deleting the row, the stale-occurrence re-read and its same-occurrence control, the seconds-to-nanoseconds parse with out-of-range values, the drop with a warning on a failed re-read, the re-read before the row is touched, the bound with a stale frame, and each entity asked exactly once. Each test failed first with its guard removed.
Gateway
ctest: 3884 tests, 0 failures.pre-commitpasses.Docker: a fault confirmed, a value flipped, the gateway restarted twice. The original frame is served with its own
captured_at. A fault confirmed while the gateway was down gets astartupframe. The same script on the base re-derives the frame. With the demo'sstart.sh, a frame captured at 95.0 °C is served afterstop.shandstart.shwhile the plant reads 25.55 °C, and the documented purge leaves an empty store.Checklist