Repository navigation
storage: cap RocksDB open table files and merge small ones (2026-10-28 archive crash loop) - #159
Merged
Merged
Conversation
2026-10-28: archives e and obs1 crash-looped at open (355 / 291 restarts) with `Too many open files` once they held ~67 000 table files against the 65 536 soft limit. WAL-full flushes write the quiet height-keyed families a few KB at a time and RocksDB only moves such files down a level, never merges them (e: anchors 22 084 files / 45 MB, perp_inputs 22 083 / 51 MB, blocks 17 542 / 22.5 GB). Pruned validators escape through compact_pruned_history; archives never prune. - max_open_files = 16 384: a table cache instead of every file held open. - Storage::tidy_small_files: per family, runs of >= 8 files averaging < 4 MiB, chunked at 256 MiB, force-compacted to the bottom level; 1 GiB per pass. - The node runs it every 1 024 blocks off the loop, under the same one-at-a-time flag as the history compaction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incident
2026-10-28 from ~10:21Z, archives e and obs1 crash-looped at RocksDB open (
Too many open files, 355 / 291 restarts): ~67 000 table files vs the 65 536 soft limit. Consensus kept going (23/25), but randscan, the public RPC and the bridge observer froze. Mitigated on the hosts withLimitNOFILE=524288:524288; both back at head 11:32Z.Cause
Default RocksDB options hold every table file open. WAL-full flushes write the quiet height-keyed families a few KB at a time, and with non-overlapping keys RocksDB only trivially moves those files down, never merges them. On e:
anchors22 084 files / 45 MB,perp_inputs22 083 / 51 MB,blocks17 542 / 22.5 GB. Each restart adds ~18 more. Pruned validators escape viacompact_pruned_history; archives never prune.Change
max_open_files = 16 384(table cache).Storage::tidy_small_files(budget): per family, in key order, chunks of ≤ 256 MiB holding ≥ 8 files averaging < 4 MiB are force-compacted to the bottom level; ≤ 1 GiB rewritten per pass (bounded temp disk — e has ~4.6 GB free).spawn_blocking, sharing thecompactingflag with the history compaction.Tests
New:
the_store_caps_its_open_table_files,tidy_merges_a_family_of_tiny_files_and_keeps_every_row(200 files → ≤ 2, all rows intact),tidy_leaves_a_family_of_few_files_alone,tidy_stops_at_its_byte_budget,should_tidy_fires_every_tidy_interval_when_idle_and_shares_the_compaction_flag. Fullcargo test -p randprotocol-nodegreen (lib 808 passed); no clippy warnings in changed files.Not measured: a tidy pass against a real archive store (first pass on e will drain the ~22 GB
blocksbacklog over ~20 passes ≈ 15 h).🤖 Generated with Claude Code