Skip to content

Report a failing sink, and back off a collection that keeps failing - #37

Merged
mikelangmayr merged 1 commit into
mainfrom
mike/keygrabber-failure-reporting
Sep 18, 2026
Merged

mikelangmayr merged 1 commit into
mainfrom
mike/keygrabber-failure-reporting

Conversation

@mikelangmayr

Copy link
Copy Markdown
Collaborator

Two bugs found by running the keygrabber against a real peer and a dead InfluxDB, while writing the hispec deployment config (hispec#216). Both are in failure handling, which is exactly what the unit tests were not exercising.

isconnected reported healthy while every write failed. Observed on a live daemon: queuedepth=25, pointswritten=0, and yet isconnected=True, writeerrors=0, lasterror=None. An operator would have seen a healthy archiver writing nothing.

The cause is a seam between two pieces that are each correct alone. RetryingWriter.write catches SinkWriteError, queues the batch and returns 0, which is what makes retrying possible; so the daemon's except LibbyError never fired and it set _sink_healthy = True regardless.

  • RetryingWriter now exposes healthy and failed_attempts, with a docstring saying why a caller cannot learn this from an exception
  • The daemon reads health back from the writer after every write and flush, and logs only on the transition, so a dead database reports once rather than every batch
  • writeerrors now counts every failed attempt including retries

A failing collection had no backoff. It logged an error every interval indefinitely, which the design called for avoiding. Measured before: twelve error lines in six seconds at a 0.5s cadence, forever, each one also rewriting lasterror.

  • A collection that fails three consecutive times doubles its interval each further failure, capped at 32x, and returns to its configured cadence on the first successful read. The cap keeps a recovered peer from waiting long to be noticed
  • The scheduler asks the collection for its interval rather than reading the config, so the backoff needs no scheduler state
  • Logging matches: first failures reported, the backoff announced once, then quiet until it answers. Measured after: three lines in six seconds, then silence

Both symptoms are now regression tests. The dead-sink one asserts through the control keywords over both transports, which is how the bug presented; the backoff ones run against a fake clock. 240 tests, pylint 10.00/10, -W docs build verified, and the docs now say how a failing sink looks across isconnected, queuedepth and pointswritten together.

@prkrtg prkrtg left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mikelangmayr
mikelangmayr merged commit 7b23b7a into main Sep 18, 2026
3 checks passed
@mikelangmayr
mikelangmayr deleted the mike/keygrabber-failure-reporting branch September 18, 2026 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants