From d1630ec1c9bd0d7bf761937e6d83656f5d89c742 Mon Sep 17 00:00:00 2001 From: Guangxue Date: Thu, 3 Sep 2026 10:47:07 +0000 Subject: [PATCH 1/2] docs: pipeline: inputs: tail: document byte completeness metrics Signed-off-by: Guangxue --- administration/monitoring.md | 2 ++ pipeline/inputs/tail.md | 27 +++++++++++++++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/administration/monitoring.md b/administration/monitoring.md index 15789fdb6..8f605c532 100644 --- a/administration/monitoring.md +++ b/administration/monitoring.md @@ -203,8 +203,10 @@ Some metrics are available only for specific plugins or runtime modes. For examp | `fluentbit_filter_records_total` | name: the name or alias for the filter instance | The number of log records this filter has ingested successfully. | counter | records | | `fluentbit_hot_reloaded_times` | hostname: the hostname on running Fluent Bit | Collect the count of hot reloaded times. | counter | times | | `fluentbit_input_bytes_total` | name: the name or alias for the input instance | The number of bytes of log records that this input instance has ingested successfully. | counter | bytes | +| `fluentbit_input_files_abandoned_bytes_total` | name: the name or alias for the input instance | The cumulative unread raw source-file bytes discarded when monitored files are terminally removed (`rotate_wait` expiration, file deletion, or truncation). Only available for the [Tail](../pipeline/inputs/tail.md) input plugin. | counter | bytes | | `fluentbit_input_files_closed_total` | name: the name or alias for the input instance | The total number of closed files. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin. | counter | files | | `fluentbit_input_files_opened_total` | name: the name or alias for the input instance | The total number of opened files. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin. | counter | files | +| `fluentbit_input_files_processed_bytes_total` | name: the name or alias for the input instance | The cumulative raw source-file bytes past which the resumable offset has advanced. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin. | counter | bytes | | `fluentbit_input_files_rotated_total` | name: the name or alias for the input instance | The total number of rotated files. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin. | counter | files | | `fluentbit_input_ingestion_paused` | name: the name or alias for the input instance | Indicates whether the input instance ingestion is currently paused (1) or not (0). | gauge | boolean | | `fluentbit_input_http_server_ingress_queue_busy_total` | name: the name or alias for the input instance | The total number of times an HTTP worker input hit a full deferred ingress queue and applied backpressure. Only available for HTTP-based inputs using `http_server.workers` greater than `1`. | counter | events | diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index 6118106fe..73f0e83aa 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -622,3 +622,30 @@ pipeline: {% endtab %} {% endtabs %} + +## Metrics + +The Tail input plugin exposes plugin-specific metrics through the internal HTTP monitoring interface. For general monitoring configuration and metric endpoint details, see [Monitoring](../../administration/monitoring.md). + +The following metrics track file lifecycle events, processing progress, and unread byte abandonment: + +| Metric | Type | Description | Unit | +| ------ | ---- | ----------- | ---- | +| `fluentbit_input_files_opened_total` | counter | The total number of opened files. | files | +| `fluentbit_input_files_closed_total` | counter | The total number of closed files. | files | +| `fluentbit_input_files_rotated_total` | counter | The total number of rotated files. | files | +| `fluentbit_input_files_processed_bytes_total` | counter | The cumulative raw source-file bytes past which the resumable offset has advanced. | bytes | +| `fluentbit_input_files_abandoned_bytes_total` | counter | The cumulative unread raw source-file bytes discarded when monitored files are terminally removed (`rotate_wait` expiration, file deletion, or truncation). | bytes | +| `fluentbit_input_long_line_skipped_total` | counter | The total number of skipped occurrences for long lines when `skip_long_lines` is enabled. | occurrences | +| `fluentbit_input_long_line_truncated_total` | counter | The total number of truncated occurrences for long lines when `truncate_long_lines` is enabled. | occurrences | +| `fluentbit_input_multiline_truncated_total` | counter | The total number of truncated occurrences for multiline messages when `multiline.parser` is configured. | occurrences | + +### Monitor log completeness + +Comparing `fluentbit_input_files_processed_bytes_total` and `fluentbit_input_files_abandoned_bytes_total` provides a lag-independent indicator of whether log data was fully drained before files were removed from rotation or deleted: + +$$ +\text{Completeness} = \frac{\text{processed\_bytes}}{\text{processed\_bytes} + \text{abandoned\_bytes}} +$$ + +Under normal operation without premature file removal or backpressure drops during rotation, `fluentbit_input_files_abandoned_bytes_total` remains `0`, yielding a completeness ratio of `1.0`. From 2a64a9f8222b4d6222899902f3225c2f57c2320f Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Thu, 3 Sep 2026 21:25:20 +0200 Subject: [PATCH 2/2] docs: pipeline: inputs: tail: clarify byte completeness metrics - Define the completeness ratio as not applicable when both byte counters are 0, and note that alerting rules must guard the division by zero - State that a ratio of 1.0 applies only when the denominator is non-zero and abandoned bytes are 0 - Correct the abandoned bytes description: bytes are counted whenever a file is truncated or removed from the monitored list, not only on rotate_wait expiration, deletion, or truncation - Reword the processed bytes description to lead with its plain meaning - Name administration/monitoring.md as the canonical metrics reference from the Tail metrics section - Keep both descriptions synchronized in administration/monitoring.md - Add rate-based PromQL example with a denominator filter, since the cumulative counters report completeness over the process lifetime and an unguarded query returns NaN when neither counter advances Signed-off-by: Eric D. Schabell --- administration/monitoring.md | 4 ++-- pipeline/inputs/tail.md | 23 ++++++++++++++++++----- 2 files changed, 20 insertions(+), 7 deletions(-) diff --git a/administration/monitoring.md b/administration/monitoring.md index 8f605c532..22b5c89c8 100644 --- a/administration/monitoring.md +++ b/administration/monitoring.md @@ -203,10 +203,10 @@ Some metrics are available only for specific plugins or runtime modes. For examp | `fluentbit_filter_records_total` | name: the name or alias for the filter instance | The number of log records this filter has ingested successfully. | counter | records | | `fluentbit_hot_reloaded_times` | hostname: the hostname on running Fluent Bit | Collect the count of hot reloaded times. | counter | times | | `fluentbit_input_bytes_total` | name: the name or alias for the input instance | The number of bytes of log records that this input instance has ingested successfully. | counter | bytes | -| `fluentbit_input_files_abandoned_bytes_total` | name: the name or alias for the input instance | The cumulative unread raw source-file bytes discarded when monitored files are terminally removed (`rotate_wait` expiration, file deletion, or truncation). Only available for the [Tail](../pipeline/inputs/tail.md) input plugin. | counter | bytes | +| `fluentbit_input_files_abandoned_bytes_total` | name: the name or alias for the input instance | The cumulative number of unread raw source-file bytes discarded when a monitored file is truncated or removed from the monitored list. Removal includes file deletion, rotation, and `rotate_wait` expiration. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin. | counter | bytes | | `fluentbit_input_files_closed_total` | name: the name or alias for the input instance | The total number of closed files. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin. | counter | files | | `fluentbit_input_files_opened_total` | name: the name or alias for the input instance | The total number of opened files. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin. | counter | files | -| `fluentbit_input_files_processed_bytes_total` | name: the name or alias for the input instance | The cumulative raw source-file bytes past which the resumable offset has advanced. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin. | counter | bytes | +| `fluentbit_input_files_processed_bytes_total` | name: the name or alias for the input instance | The cumulative number of raw source-file bytes processed. This counter advances with the resumable offset, so bytes held in the buffer as an incomplete trailing line aren't counted until that line completes. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin. | counter | bytes | | `fluentbit_input_files_rotated_total` | name: the name or alias for the input instance | The total number of rotated files. Only available for the [Tail](../pipeline/inputs/tail.md) input plugin. | counter | files | | `fluentbit_input_ingestion_paused` | name: the name or alias for the input instance | Indicates whether the input instance ingestion is currently paused (1) or not (0). | gauge | boolean | | `fluentbit_input_http_server_ingress_queue_busy_total` | name: the name or alias for the input instance | The total number of times an HTTP worker input hit a full deferred ingress queue and applied backpressure. Only available for HTTP-based inputs using `http_server.workers` greater than `1`. | counter | events | diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index 73f0e83aa..f341e2242 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -625,17 +625,17 @@ pipeline: ## Metrics -The Tail input plugin exposes plugin-specific metrics through the internal HTTP monitoring interface. For general monitoring configuration and metric endpoint details, see [Monitoring](../../administration/monitoring.md). +The Tail input plugin exposes plugin-specific metrics through the internal HTTP monitoring interface. The following table lists the metrics that apply to this plugin. [Monitoring](../../administration/monitoring.md) remains the complete reference for Fluent Bit metrics, including the labels attached to each one and the endpoints that expose them. -The following metrics track file lifecycle events, processing progress, and unread byte abandonment: +These metrics track file lifecycle events, processing progress, and unread byte abandonment: | Metric | Type | Description | Unit | | ------ | ---- | ----------- | ---- | | `fluentbit_input_files_opened_total` | counter | The total number of opened files. | files | | `fluentbit_input_files_closed_total` | counter | The total number of closed files. | files | | `fluentbit_input_files_rotated_total` | counter | The total number of rotated files. | files | -| `fluentbit_input_files_processed_bytes_total` | counter | The cumulative raw source-file bytes past which the resumable offset has advanced. | bytes | -| `fluentbit_input_files_abandoned_bytes_total` | counter | The cumulative unread raw source-file bytes discarded when monitored files are terminally removed (`rotate_wait` expiration, file deletion, or truncation). | bytes | +| `fluentbit_input_files_processed_bytes_total` | counter | The cumulative number of raw source-file bytes processed. This counter advances with the resumable offset, so bytes held in the buffer as an incomplete trailing line aren't counted until that line completes. | bytes | +| `fluentbit_input_files_abandoned_bytes_total` | counter | The cumulative number of unread raw source-file bytes discarded when a monitored file is truncated or removed from the monitored list. Removal includes file deletion, rotation, and `rotate_wait` expiration. | bytes | | `fluentbit_input_long_line_skipped_total` | counter | The total number of skipped occurrences for long lines when `skip_long_lines` is enabled. | occurrences | | `fluentbit_input_long_line_truncated_total` | counter | The total number of truncated occurrences for long lines when `truncate_long_lines` is enabled. | occurrences | | `fluentbit_input_multiline_truncated_total` | counter | The total number of truncated occurrences for multiline messages when `multiline.parser` is configured. | occurrences | @@ -648,4 +648,17 @@ $$ \text{Completeness} = \frac{\text{processed\_bytes}}{\text{processed\_bytes} + \text{abandoned\_bytes}} $$ -Under normal operation without premature file removal or backpressure drops during rotation, `fluentbit_input_files_abandoned_bytes_total` remains `0`, yielding a completeness ratio of `1.0`. +The denominator is zero when both counters are `0`, which is the case at startup before Fluent Bit processes or abandons any bytes. Treat the ratio as not applicable in that case instead of as a failure. Alerting rules must guard against this division by zero. + +When the denominator is non-zero and `fluentbit_input_files_abandoned_bytes_total` is `0`, the ratio is `1.0`. This is the expected value under normal operation without premature file removal or backpressure drops during rotation. + +Both metrics are cumulative counters, so this formula reports completeness across the entire lifetime of the Fluent Bit process. A single abandonment event lowers the value permanently, even after the underlying problem is fixed. To measure completeness over a recent window instead, apply the same formula to the rate of each counter: + +```text +sum(rate(fluentbit_input_files_processed_bytes_total[5m])) +/ +(sum(rate(fluentbit_input_files_processed_bytes_total[5m])) + + sum(rate(fluentbit_input_files_abandoned_bytes_total[5m])) > 0) +``` + +The trailing `> 0` drops the result when the denominator is zero, which happens when both counters exist but neither advances during the window. Without that filter, the query evaluates `0 / 0` and returns a NaN sample instead of an empty result.