feat(gateway): keep a plugin entity's freeze frame across a gateway restart

docs/config/server.rst

@@ -384,6 +384,23 @@ Configure how the gateway connects to the fault manager services and event topic
        values, marked ``connected: false`` in the snapshot's ``x-medkit`` block.
        Explicit snapshot config in the fault manager always wins when present.
        Only active when plugins are loaded.
+   * - ``entity_freeze_frame.storage.path``
+     - string
+     - ``""``
+     - SQLite file the captured frames are persisted in, so a restart serves the
+       values frozen at fault time instead of re-reading the plant. When empty,
+       the frames go in ``entity_freeze_frames.db`` next to
+       ``triggers.storage.path``. With that empty too they stay in memory and are
+       lost on restart. A reloaded frame keeps its original ``captured_at`` and
+       its ``capture_origin``, and the startup catch-up then runs only for faults
+       that have no stored frame. The retained-frame bound of 256 faults counts
+       reloaded and freshly captured frames together, dropping the oldest first,
+       and a frame whose fault the fault manager no longer holds at all is
+       dropped at startup. A frame belonging to an occurrence that has since
+       been cleared and confirmed again is re-read at startup and marked
+       ``capture_origin: startup`` rather than served as the current one. When
+       that re-read cannot answer, the stale frame is discarded with a warning
+       naming the fault code and the entity.
 
 When ``fault_manager.namespace`` is set, the gateway also subscribes to the matching
 fault event topic (for example ``/robot1/fault_manager/events`` instead of the default

docs/tutorials/snapshots.rst

@@ -201,15 +201,37 @@ and may predate the confirmation by the length of the outage; the entry's
 payload includes one, ``source_timestamp`` (the payload's own timestamp)
 alongside ``captured_at``.
 
+With ``entity_freeze_frame.storage.path`` set, a captured frame survives a
+gateway restart: it is reloaded at start and served exactly as it was
+captured, with its original ``captured_at`` and no ``capture_origin`` marker.
+Set the path to a file on a volume that outlives the container, or leave it
+empty and the frames go next to the trigger store
+(``triggers.storage.path``). With neither set they are process memory only
+and a restart loses them. A plugin entity keeps exactly one frame per fault: a
+re-confirm re-samples the plugin and replaces it, on disk as in memory, and a
+re-confirm the gateway was down for is re-read at startup and marked
+``capture_origin: startup``, so a confirmed occurrence never serves the
+previous one's values. The startup comparison is against the fault's
+``first_occurred``, which the fault manager resets on reactivation from
+``CLEARED``, so it holds for exactly the faults the catch-up sees: a fault
+that re-failed but has not re-confirmed yet is not in the confirmed list and
+keeps its frame until it does confirm, and a ``HEALED`` to ``FAILED`` cycle
+does not reset ``first_occurred`` at all (healing is off by default). If the
+re-read cannot answer, because the entity is unreachable or serves nothing
+usable, the stale frame is discarded rather than served, and the gateway warns
+with the fault code and the entity so the missing evidence is not silent.
+
 Faults that are already confirmed when the gateway starts are caught up at
 startup: the gateway lists the confirmed faults and captures a frame for each
-plugin-backed one, so a device standing in fault across a gateway restart
-still gets a frame. Catch-up frames carry ``"capture_origin": "startup"`` in
-their ``x-medkit`` block because their values were read at gateway start, not
-when the fault confirmed (which may be long before, since the fault manager
+plugin-backed one that does not already have a stored frame, so a device
+standing in fault across a gateway restart still gets a frame, and one whose
+frame was already taken keeps the values from its own confirm edge instead of
+today's. Catch-up frames carry ``"capture_origin": "startup"`` in their
+``x-medkit`` block because their values were read at gateway start, not when
+the fault confirmed (which may be long before, since the fault manager
 persists faults); ``captured_at`` always stamps the moment the values were
-read. Frames without the marker were captured on the confirm edge. Disable
-with:
+read. Frames without the marker were captured on the confirm edge, and a
+reloaded frame keeps whichever marker it was captured with. Disable with:
 
 .. code-block:: bash
 

src/ros2_medkit_gateway/CHANGELOG.rst

@@ -2,6 +2,11 @@
 Changelog for package ros2_medkit_gateway
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+Forthcoming
+-----------
+* Entity freeze-frames survive a restart when ``entity_freeze_frame.storage.path`` names a database. Without one the frames lived in process memory, so a restart threw them away and the startup catch-up re-read the plant as it is now, serving today's values under the original fault and marking them ``x-medkit.capture_origin: startup`` - the values at fault time, which are the point of a freeze-frame, were gone. A reloaded frame is served exactly as it was captured: its original ``captured_at``, its own ``capture_origin`` (absent on a confirm-edge frame), and the ``connected`` / ``source_timestamp`` provenance it carried, and the startup catch-up then re-reads only the faults that have no frame. Leaving the path empty puts the store in ``entity_freeze_frames.db`` next to ``triggers.storage.path``. With neither set the frames stay in memory as before, and a store that cannot be opened or written is reported while the capture keeps working. The retained-frame bound of 256 faults counts reloaded and freshly captured frames together, a frame whose fault the fault_manager no longer holds in any status is dropped at startup (one reported as cleared keeps its frame), and a frame belonging to an occurrence that has since been cleared and re-confirmed is re-read at startup and marked ``capture_origin: startup`` rather than served as the current one
+* Contributors: @bburda
+
 0.7.0 (2026-08-27)
 ------------------
 * Rosbag bulk-data is addressed by recording id instead of fault code, so a fault holding several recordings can expose each one. ``GET /{entity}/bulk-data/rosbags`` now emits one descriptor per recording rather than one per fault - a burst that shares a bag used to appear as several entries each reporting the full bag size - and the covered faults move into ``x-medkit.fault_codes`` (was the scalar ``x-medkit.fault_code``). Old URLs keep working: an id that is not a recording is resolved as a fault code and serves that fault's newest recording, which is what it returned before. Authorization is unchanged in effect - a download is allowed when any fault the recording covers is in the entity's source scope, which is exactly the set that could reach it previously (`#623 <https://github.com/selfpatch/ros2_medkit/pull/623>`_, `#620 <https://github.com/selfpatch/ros2_medkit/issues/620>`_)

src/ros2_medkit_gateway/CMakeLists.txt

@@ -840,6 +840,11 @@ if(BUILD_TESTING)
   target_link_libraries(test_entity_freeze_frame_capture gateway_ros2)
   medkit_target_dependencies(test_entity_freeze_frame_capture rclcpp ros2_medkit_msgs)
 
+  # Entity freeze-frame persistence (both store backends). Links gateway_core
+  # only (ROS-neutral).
+  medkit_add_gtest(test_entity_freeze_frame_store test/test_entity_freeze_frame_store.cpp)
+  target_link_libraries(test_entity_freeze_frame_store gateway_core)
+
   # Add update manager tests
   medkit_add_gtest(test_update_manager test/test_update_manager.cpp)
   target_link_libraries(test_update_manager gateway_ros2)

src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/entity_freeze_frame_store.hpp

@@ -0,0 +1,120 @@
+// Copyright 2026 bburda
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <map>
+#include <mutex>
+#include <string>
+#include <vector>
+
+#include <nlohmann/json.hpp>
+#include <tl/expected.hpp>
+
+namespace ros2_medkit_gateway {
+
+/// One persisted entity freeze-frame: the row shape of the store, keyed by
+/// (fault_code, entity_id).
+///
+/// `frame` holds everything that is not already a column of its own - the
+/// compact {resource_id: value} dict under "values", plus the payload
+/// provenance the capture recorded ("connected", "source_timestamp") when the
+/// plugin reported it. Keeping those in the blob rather than in columns is
+/// what lets a reloaded frame be served byte for byte as it was captured.
+struct StoredEntityFreezeFrame {
+  std::string fault_code;
+  std::string entity_id;
+  nlohmann::json frame;  ///< {"values": {...}, "connected"?: bool, "source_timestamp"?: any}
+  int64_t captured_at_ns{0};
+  std::string source;          ///< capture path that read the values
+  std::string capture_origin;  ///< "startup" for a catch-up frame, empty on a confirm edge
+};
+
+/// Persistence for the gateway's entity freeze-frames.
+///
+/// The gateway's frames are process memory, so a restart re-derives them from
+/// whatever the plant reads *now* - the values at fault time are gone and the
+/// re-read is stamped with the restart. This store is what makes the captured
+/// frame outlive the process.
+///
+/// Writes are per fault code and wholesale: a re-confirm replaces every row
+/// for that code, mirroring the in-memory map, whose entry for a code is
+/// likewise replaced as a unit. Implementations must be thread-safe.
+class EntityFreezeFrameStore {
+ public:
+  virtual ~EntityFreezeFrameStore() = default;
+
+  /// Replace every row for @p fault_code with @p frames (one row per entity).
+  /// An empty vector leaves no rows for the code.
+  virtual tl::expected<void, std::string> replace_frames(const std::string & fault_code,
+                                                         const std::vector<StoredEntityFreezeFrame> & frames) = 0;
+
+  /// Drop every row for @p fault_code. Removing a code that has no rows is
+  /// not an error: the caller evicts by code and does not track what is on disk.
+  virtual tl::expected<void, std::string> erase_frames(const std::string & fault_code) = 0;
+
+  /// Every row, oldest capture first. The order is what lets a caller honour a
+  /// retained-frame bound by keeping the newest codes.
+  virtual tl::expected<std::vector<StoredEntityFreezeFrame>, std::string> load_all() = 0;
+};
+
+/// In-memory backend: the store contract without a file, for tests and for
+/// callers that want the interface without persistence.
+class InMemoryEntityFreezeFrameStore : public EntityFreezeFrameStore {
+ public:
+  tl::expected<void, std::string> replace_frames(const std::string & fault_code,
+                                                 const std::vector<StoredEntityFreezeFrame> & frames) override {
+    std::lock_guard<std::mutex> lock(mutex_);
+    if (frames.empty()) {
+      rows_.erase(fault_code);
+      return {};
+    }
+    rows_[fault_code] = frames;
+    return {};
+  }
+
+  tl::expected<void, std::string> erase_frames(const std::string & fault_code) override {
+    std::lock_guard<std::mutex> lock(mutex_);
+    rows_.erase(fault_code);
+    return {};
+  }
+
+  tl::expected<std::vector<StoredEntityFreezeFrame>, std::string> load_all() override {
+    std::lock_guard<std::mutex> lock(mutex_);
+    std::vector<StoredEntityFreezeFrame> all;
+    for (const auto & entry : rows_) {
+      all.insert(all.end(), entry.second.begin(), entry.second.end());
+    }
+    // Same total order the SQLite backend serves, so a caller's bound-keeping
+    // behaves identically on both.
+    std::stable_sort(all.begin(), all.end(), [](const StoredEntityFreezeFrame & a, const StoredEntityFreezeFrame & b) {
+      if (a.captured_at_ns != b.captured_at_ns) {
+        return a.captured_at_ns < b.captured_at_ns;
+      }
+      if (a.fault_code != b.fault_code) {
+        return a.fault_code < b.fault_code;
+      }
+      return a.entity_id < b.entity_id;
+    });
+    return all;
+  }
+
+ private:
+  mutable std::mutex mutex_;
+  std::map<std::string, std::vector<StoredEntityFreezeFrame>> rows_;
+};
+
+}  // namespace ros2_medkit_gateway

src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/sqlite_entity_freeze_frame_store.hpp

@@ -0,0 +1,62 @@
+// Copyright 2026 bburda
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#pragma once
+
+#include <sqlite3.h>
+
+#include <mutex>
+#include <string>
+#include <vector>
+
+#include "ros2_medkit_gateway/core/entity_freeze_frame_store.hpp"
+
+namespace ros2_medkit_gateway {
+
+/// SQLite-backed entity freeze-frame persistence.
+///
+/// Thread-safe via internal mutex. The table is created on first open. Use
+/// ":memory:" for an ephemeral database.
+class SqliteEntityFreezeFrameStore : public EntityFreezeFrameStore {
+ public:
+  /// Open (or create) the database at `db_path`.
+  /// @throws std::runtime_error on SQLite open/init failure.
+  explicit SqliteEntityFreezeFrameStore(const std::string & db_path);
+
+  ~SqliteEntityFreezeFrameStore() override;
+
+  // Non-copyable, non-movable (owns SQLite connection)
+  SqliteEntityFreezeFrameStore(const SqliteEntityFreezeFrameStore &) = delete;
+  SqliteEntityFreezeFrameStore & operator=(const SqliteEntityFreezeFrameStore &) = delete;
+  SqliteEntityFreezeFrameStore(SqliteEntityFreezeFrameStore &&) = delete;
+  SqliteEntityFreezeFrameStore & operator=(SqliteEntityFreezeFrameStore &&) = delete;
+
+  tl::expected<void, std::string> replace_frames(const std::string & fault_code,
+                                                 const std::vector<StoredEntityFreezeFrame> & frames) override;
+  tl::expected<void, std::string> erase_frames(const std::string & fault_code) override;
+  tl::expected<std::vector<StoredEntityFreezeFrame>, std::string> load_all() override;
+
+ private:
+  /// Create the table if it does not exist.
+  void initialize_schema();
+
+  /// Delete every row for a code. Caller holds mutex_.
+  tl::expected<void, std::string> delete_code_locked(const std::string & fault_code);
+
+  std::string db_path_;
+  sqlite3 * db_{nullptr};
+  mutable std::mutex mutex_;
+};
+
+}  // namespace ros2_medkit_gateway