fix(gateway): download a split recording's first file, and say how many there are

docs/api/rest.rst

@@ -1758,7 +1758,8 @@ List all bulk-data items in a category for the entity.
            "fault_codes": ["MOTOR_OVERHEAT", "MOTOR_STALL"],
            "duration_sec": 6.0,
            "format": "mcap",
-           "recording_id": "fault_MOTOR_OVERHEAT_1738664999000"
+           "recording_id": "fault_MOTOR_OVERHEAT_1738664999000",
+           "storage_files": 1
          }
        }
      ]
@@ -1772,6 +1773,13 @@ recording therefore reports its size once. One fault code can appear on several
 descriptors, one per occurrence it kept, told apart by ``creation_date``, which
 is the time that recording was made.
 
+``x-medkit.storage_files`` is how many storage files the recording is held in,
+read from the recording's own metadata. A ``1`` is the normal case and says the
+download below hands over the whole recording. A larger number is a split
+recording, and then the download hands over the first of those files. The field
+is omitted when the gateway cannot read the recording's metadata, so an absent
+field means "not known here" rather than "one".
+
 ``size`` is the number of bytes the download route below puts on the wire for
 that descriptor, so a client can size a buffer or a progress bar from the
 listing. For a rosbag held in a single storage file, which is the normal case,
@@ -1792,12 +1800,28 @@ split across several storage files, past the configured maximum bag size: the
 download can hand over only one of them, no single file describes the transfer,
 and the API reports the recording's total instead.
 
-For that split case the three numbers stop agreeing, and deliberately so. The
-descriptor ``size`` and the nested ``size_bytes`` report the recording's total
-while the download's ``Content-Length`` is the one storage file it hands over,
-so ``size`` exceeds ``Content-Length``. That gap is the signal: a client that
-compares the two can tell the transfer it just made is a part of the recording
-rather than the whole of it, which no single reported number could express.
+**A split recording.** Past the configured maximum bag size a recording is held
+in several storage files rather than one. Three things are then true at once,
+and they are meant to be read together:
+
+- the download hands over the **first storage file the recording's own metadata
+  names that is on disk**. Normally that is the first segment, where the
+  recording starts. A named segment that is missing from disk is skipped in
+  favour of the next one that is there, and when none of them is on disk the
+  download fails rather than serving some other file that happens to sit in the
+  bag directory. Which file is served does not depend on the host's filesystem
+  and does not change between two requests for the same recording,
+- ``x-medkit.storage_files`` says **how many storage files the metadata names**,
+  so a client can tell it received a part and know how many parts the recording
+  was recorded in,
+- ``size`` stays the **whole recording**, as does the nested
+  ``environment_data.snapshots[].size_bytes``, while the download's
+  ``Content-Length`` is that one file, so ``size`` exceeds ``Content-Length``.
+
+That last gap is a second signal for the same fact and is kept because a client
+that has already made the transfer can read it without listing anything. The
+storage files of a split recording after the first are not addressable over the
+API today.
 
 Download Bulk Data
 ~~~~~~~~~~~~~~~~~~
@@ -1814,8 +1838,10 @@ Download a specific bulk-data file.
   a pre-#620 fault-code URL is not the segment the client sent, and the format
   is the one persisted at capture time (``mcap`` or ``sqlite3``). For every
   other category it is the stored item's own name, e.g. ``report.zip``.
-- ``Content-Length``: the served file's length. For how it relates to the
-  descriptor ``size`` of the same recording, see
+- ``Content-Length``: the served file's length. For a rosbag that is the
+  recording's single storage file, or, when the recording is split, the first
+  of its storage files. For how it relates to the descriptor ``size`` and
+  ``x-medkit.storage_files`` of the same recording, see
   :ref:`One recording, one size <rest-recording-size-rule>`
 - ``Accept-Ranges``: ``bytes`` - the download is served by a range-aware
   provider, so a client may fetch part of the file

src/ros2_medkit_gateway/CHANGELOG.rst

@@ -2,6 +2,11 @@
 Changelog for package ros2_medkit_gateway
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+Forthcoming
+-----------
+* Downloading a rosbag recording that is split across several storage files, past ``snapshots.rosbag.max_bag_size_mb``, now hands over the first storage file the recording's own ``metadata.yaml`` names that is on disk, normally the first segment, where the recording starts. A named segment missing from disk is skipped in favour of the next one that is there, and when none of them is on disk the download fails rather than serving another file. Previously the gateway walked the bag directory and served whichever ``.db3`` or ``.mcap`` it yielded first, so which segment a client received was decided by the host's filesystem and could differ between two requests for the same recording, and a file the recording never named could be served under its id. A bag directory is now read only when the recording will not say what it holds, meaning no ``metadata.yaml``, one that cannot be read or parsed, or a ``relative_file_paths`` naming nothing. The rosbag descriptor gains ``x-medkit.storage_files``, how many storage files the recording's metadata names, so a client can tell a whole recording from one part of a split one. It is omitted in exactly those cases where the metadata does not answer, so an absent field means "not known here" rather than "one". The descriptor ``size`` is unchanged and still reports the whole recording for a split, which is why it exceeds the download's ``Content-Length``.
+* The ``size`` a rosbag descriptor reports under ``GET /{entity}/bulk-data/rosbags`` is the storage file the download serves, measured through the same resolver the download uses, instead of the bag directory's total. The two differ by ``metadata.yaml``, which is never served, so every listing used to overstate its own download, on a short recording by around a tenth of the transfer, and a client sizing a buffer or a progress bar from the listing never reached the end. A bag whose ``metadata.yaml`` names exactly one storage file now resolves to that file rather than to whichever ``.db3`` or ``.mcap`` the directory yielded first, so a stray file beside the recording can no longer be served and sized in place of the real one, and the gateway and the fault manager decide it from the same field. A recording split across several storage files keeps the whole recording's figure, which is what the fault manager reports for it. A bag directory the gateway cannot read now costs its own row rather than the whole request: the resolver used the throwing filesystem calls, so one unreadable directory, or one removed by quota eviction mid-walk, answered ``500`` and took every other recording of that entity out of the listing with it. The gateway README no longer documents ``GET /faults/{code}/snapshots`` and ``.../snapshots/bag``, which have answered ``404`` since ``0.2.0``, because snapshots are returned inline with the fault and recordings are downloaded through the bulk-data endpoints.
+
 0.7.0 (2026-08-27)
 ------------------
 * Rosbag bulk-data is addressed by recording id instead of fault code, so a fault holding several recordings can expose each one. ``GET /{entity}/bulk-data/rosbags`` now emits one descriptor per recording rather than one per fault - a burst that shares a bag used to appear as several entries each reporting the full bag size - and the covered faults move into ``x-medkit.fault_codes`` (was the scalar ``x-medkit.fault_code``). Old URLs keep working: an id that is not a recording is resolved as a fault code and serves that fault's newest recording, which is what it returned before. Authorization is unchanged in effect - a download is allowed when any fault the recording covers is in the entity's source scope, which is exactly the set that could reach it previously (`#623 <https://github.com/selfpatch/ros2_medkit/pull/623>`_, `#620 <https://github.com/selfpatch/ros2_medkit/issues/620>`_)

src/ros2_medkit_gateway/README.md

@@ -1165,10 +1165,13 @@ download serves one storage file, verbatim, named `<recording_id>.<format>`
 For a recording held in a single storage file, which is the normal case, the
 descriptor `size` is that file's length and therefore the length of the
 download. A recording that grew past `snapshots.rosbag.max_bag_size_mb` is split
-across several storage files and the download hands over only one of them. The
-descriptor then reports the recording's total, so `size` exceeds the download's
-`Content-Length`. See [the size rule](../../docs/api/rest.rst) in the REST API
-reference for the full statement.
+across several storage files, and then the download hands over the first one the
+recording's own metadata names that is on disk (a named file missing from disk is
+skipped, and if none is there the download fails rather than serving another
+file), `x-medkit.storage_files` says how many the metadata names, and `size`
+stays the whole recording, so it exceeds the download's `Content-Length`. See
+[the size rule](../../docs/api/rest.rst) in the REST API reference for the full
+statement.
 
 **Rosbag Configuration:**
 

src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/http/handlers/bulkdata_handlers.hpp

@@ -14,6 +14,7 @@
 
 #pragma once
 
+#include <cstddef>
 #include <cstdint>
 #include <optional>
 #include <string>
@@ -123,11 +124,32 @@ class BulkDataHandlers {
    * For a recording split across several files the listing does not come through
    * here at all: it carries the recording's total from the fault manager while
    * this route still hands over one file, and the gap is what tells a client the
-   * transfer is partial. See the size rule in ``docs/api/rest.rst``.
+   * transfer is partial, alongside ``x-medkit.storage_files`` in the descriptor.
+   * See the size rule in ``docs/api/rest.rst``.
    *
-   * When the bag's own ``metadata.yaml`` names exactly one storage file and that
-   * file exists, that is the file. Otherwise the directory is scanned for the
-   * first ``.db3`` or ``.mcap`` in whatever order it yields.
+   * The bag's own ``metadata.yaml`` decides, and when it has decided nothing
+   * else gets a vote. The answer is the first file named in
+   * ``relative_file_paths`` that is on disk. That order is the capture order, so
+   * for a split recording the first name is where the recording begins and a
+   * client that fetches one gets its start rather than an arbitrary slice. A
+   * name that is not on disk is skipped, because a half-copied bag leaves
+   * metadata naming a file that is gone and failing the request there would cost
+   * a recording whose other segments are readable. A name that is absolute or
+   * climbs through ``..`` is skipped as well: it would resolve outside the bag
+   * directory and put a file that is not part of the recording on the wire.
+   *
+   * When the bag named files and none of them is on disk the answer is the empty
+   * string. Falling through to the directory there served whatever ``.db3`` or
+   * ``.mcap`` sat beside the recording, under this recording's id and against a
+   * ``storage_files`` count the served file is not a member of.
+   *
+   * Only when the bag will not say what it holds - no ``metadata.yaml``, one
+   * this process cannot read or parse, or a ``relative_file_paths`` naming
+   * nothing - is the directory scanned for the first ``.db3`` or ``.mcap`` in
+   * whatever order it yields. That is the same set of shapes
+   * ``detail::rosbag_storage_file_count`` declines, which is the invariant: the
+   * count and the served file are both read from the metadata, or both from the
+   * directory, never one from each.
    *
    * @param path Path to rosbag (can be file or directory)
    * @return Resolved file path, or empty string if not found
@@ -226,6 +248,47 @@ std::vector<std::string> rosbag_attached_fault_codes(const nlohmann::json & rosb
  */
 bool rosbag_resolved_by_fault_code(const nlohmann::json & rosbag_data, const std::string & requested_id);
 
+/**
+ * @brief How many storage files one recording is held in.
+ *
+ * Read from the bag's own ``metadata.yaml``, the same field
+ * ``BulkDataHandlers::resolve_rosbag_file_path`` picks the served file out of,
+ * so the count and the choice of segment cannot describe different recordings.
+ * A @p bag_path that is itself a storage file is one by definition and carries
+ * no metadata beside it under that name to consult.
+ *
+ * It reaches the client as ``x-medkit.storage_files`` on the rosbag descriptor,
+ * and what it is for is the split case. There the descriptor ``size`` is the
+ * whole recording while the download hands over one segment, so ``size`` and
+ * ``Content-Length`` differ. Without this field that difference has no stated
+ * reason, and a client holding one segment has no way to learn that the rest of
+ * the recording exists. A ``1`` says the transfer was the whole recording.
+ *
+ * nullopt when the bag will not say - no metadata, unreadable metadata, not the
+ * shape rosbag2 writes, or a ``relative_file_paths`` naming nothing - and the
+ * field is then omitted from the descriptor rather than defaulted. Counting the
+ * directory's ``.db3`` / ``.mcap`` files instead would count a stray beside the
+ * recording, and defaulting to one would claim a recording is whole on the
+ * evidence of nothing. A list naming nothing is declined rather than reported as
+ * zero for the same reason: the recording is not empty, the bag did not answer.
+ *
+ * The set of shapes declined here is exactly the set on which
+ * ``BulkDataHandlers::resolve_rosbag_file_path`` falls back to the directory, so
+ * the count and the served file are read from the metadata together or from the
+ * directory together. A count taken from one source describing a file chosen by
+ * the other is the state this pairing exists to make unreachable.
+ *
+ * Never throws, for the same reason as the two helpers around it: it runs once
+ * per row of a listing, and one unreadable recording must not cost the entity's
+ * other recordings.
+ *
+ * @param bag_path Bag path as stored by the fault manager. A bag directory, or
+ *                 a bare storage file, which both answer
+ * @return The number of storage files the recording names, or nullopt when the
+ *         bag's metadata cannot be read
+ */
+std::optional<std::size_t> rosbag_storage_file_count(const std::string & bag_path);
+
 /**
  * @brief Bytes a rosbag download puts on the wire for one recording.
  *
@@ -244,8 +307,8 @@ bool rosbag_resolved_by_fault_code(const nlohmann::json & rosbag_data, const std
  * figure. That covers a bag this process cannot see or read at all, and it
  * covers a recording split across several storage files past the configured
  * maximum bag size: the download hands over one segment, so no single file is
- * the transfer, and answering with whichever segment the resolver reached first
- * advertised a split recording at the size of one part of it. The row's figure
+ * the transfer, and answering with the segment it hands over advertised a split
+ * recording at the size of one part of it. The row's figure
  * is the fault manager's answer to the same question, decided from the same
  * ``metadata.yaml``, so deferring to it keeps the two API surfaces agreeing on
  * one recording.
@@ -281,6 +344,10 @@ std::optional<uint64_t> rosbag_served_bytes(const std::string & bag_path);
  * process cannot see keeps the row's own figure: it is the only number left,
  * and a recording listed with a zero size reads as an empty one.
  *
+ * ``x-medkit.storage_files`` carries how many files the recording is held in
+ * (see ``rosbag_storage_file_count``), and is omitted for a bag whose metadata
+ * this process cannot read.
+ *
  * @param rows Rosbag rows as returned by the fault manager
  * @param faults_by_code Faults keyed by code, for timestamp enrichment
  * @return One descriptor per distinct recording

src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/bulkdata.hpp

@@ -56,13 +56,18 @@ inline constexpr std::string_view dto_name<BulkDataCategoryList> = "BulkDataCate
 //   name            - human-readable filename / label (required)
 //   mimetype        - MIME type of the file (required)
 //   size            - byte count the download route serves for this item
-//                     (required). For a rosbag that is the bag's single
-//                     storage file, not the bag directory's total
+//                     (required). For a rosbag held in one storage file that is
+//                     the file, not the bag directory's total. For one split
+//                     across several it is the whole recording, so it exceeds
+//                     what the download sends, which is one storage file
 //   creation_date   - ISO 8601 timestamp string (required)
 //   description     - optional human-readable description
-//   x-medkit        - optional open vendor extension object; for rosbags:
-//                     {fault_code, duration_sec, format}; for user uploads:
-//                     arbitrary metadata JSON object set by the uploader.
+//   x-medkit        - optional open vendor extension object. For rosbags:
+//                     {fault_codes, duration_sec, format, recording_id,
+//                     storage_files}, the last of which is how many storage
+//                     files the recording's metadata names and is omitted when
+//                     that metadata cannot be read. For user uploads: an arbitrary
+//                     metadata JSON object set by the uploader.
 // =============================================================================
 struct BulkDataDescriptor {
   std::string id;