Skip to content

fix(encoding): use +Inf as the implicit histogram overflow bucket bound - #343

Open
gheorghitamutu wants to merge 1 commit into
prometheus:masterfrom
gheorghitamutu:fix/inf-overflow-bucket-bound
Open

fix(encoding): use +Inf as the implicit histogram overflow bucket bound#343
gheorghitamutu wants to merge 1 commit into
prometheus:masterfrom
gheorghitamutu:fix/inf-overflow-bucket-bound

Conversation

@gheorghitamutu

Copy link
Copy Markdown

What

Histogram::new appends f64::MAX as the sentinel for the implicit overflow bucket, and only
encoding::text translates it back to le="+Inf". Both protobuf encoders ship it verbatim, so
consumers receive an overflow bucket whose upper bound is 1.7976931348623157e+308 — a very large
finite bound, not an infinity.

This changes the sentinel to f64::INFINITY rather than translating it in each encoder.
Histogram::new is the only construction site (new_classic_and_native and
HistogramWithExemplars::new both delegate to it), so one line covers classic, exemplar and
classic+native histograms; the protobuf encoders then need no special case, and the text encoder
only changes what it compares against.

Why it matters

1. Quantiles above the ladder return ~1e308. Prometheus recognises the +Inf bucket with
IsInf, so it never saw one: it ingested the sentinel as an ordinary finite bucket and synthesised
its own le="+Inf" series from sample_count. promql's bucketQuantile never interpolates into
the +Inf bucket and returns the second-highest bound instead — the sentinel. Every quantile above
the histogram's real ladder therefore returned ~1.1e308 instead of a value. Measured on 0.25.0
scraped by Grafana Alloy into Mimir, histogram_quantile(0.99, ...) returned
1.0968850044231772e+308, which Grafana renders as "5.58e+300 years". It is silent: the series
scrapes cleanly and the panel only goes wrong once a quantile saturates. Scraping protobuf is not
optional for anyone using native histograms, since that is the only exposition offering them.

2. +Inf observations were counted in no bucket. observe_classic routes NaN to the last
bucket explicitly, but everything else through find(|(upper_bound, _)| upper_bound >= &v).
Since f64::MAX >= f64::INFINITY is false, an infinite observation incremented sum and count
while incrementing no bucket at all, leaving the overflow bucket short of _count. Prometheus's
synthetic +Inf series masked that too. Fixing only the encoders would have made this short count
visible rather than fixing it.

Why +Inf is the correct value

  • metrics.proto documents this field as holding a "+Inf bucket" (optional, inclusive).
  • client_golang writes math.Inf(1) whenever it emits that bucket, and Prometheus branches on
    math.IsInf.
  • The OpenMetrics spec requires a Histogram MetricPoint to have a bucket with an +Inf threshold,
    so the OpenMetrics encoder was in outright violation.
  • prost encodes double as a raw IEEE-754 fixed64, so infinity is bit-exact on the wire
    (00 00 00 00 00 00 F0 7F) and decodes back to +Inf on the Go side.

The native-histogram positive_upper_bound returning f64::MAX is deliberately left alone — it
mirrors client_golang's getBound and is not on any wire path.

Testing

Adds a regression test asserting that every observation, including +Inf and NaN, is counted in
some bucket. It fails before this change with
buckets=[(1.0, 1), (2.0, 0), (1.7976931348623157e308, 1)] count=3.

cargo fmt --all -- --check is clean, cargo clippy --locked --workspace --all-targets -- -D warnings
reports nothing, and cargo test --locked --all --all-features passes 91/91 with the Python
prometheus-client dependency from CONTRIBUTING.md installed.

Signed-off-by: Gheorghita MUTU <gheorghitamutu@gmail.com>
@gheorghitamutu

Copy link
Copy Markdown
Author

@mxinden @brancz I didn't create an issue for this fix as I think is quite straightforward. Do let me know if it requires one, please.

Thanks you!

@krisztianfekete krisztianfekete left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. One follow-up (can be a separate PR if you'd like as it's pre-existent): client_golang strips a trailing +Inf from user-supplied bounds since the bucket is implicit, but Histogram::new([1.0, f64::INFINITY]) now emits two le="+Inf" series.

Filtering infinite bounds in Histogram::new before appending the implicit one would fix this. Let me know if you'd like to include this one here, or if you'd open a new PR/issue for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants