Snapshot Engine + flatKV integration: phase 1

[cody-littley]

Describe your changes and provide context

Integrate flatKV and the new snapshot engine.

As part of the pipelining refactor, our goal is to move three things off of the main execution thread:

• hashing
• pebble checkpointing (i.e. on-disk snapshots)
• flushing data down into pebble

In order to limit the size of this PR, this PR only moves flushing of data off the main thread, although it does lay the groundwork for moving hashing and checkpointing off-thread.

[cursor]

PR Summary

High Risk
Changes the core multistore commit/flush contract and replaces Pebble-level caching with the snapshot engine—bugs could double-commit heights, lose writes, or break flatkv_only chains; large diff with deleted cache code increases integration risk.

Overview
Moves flatKV read caching and async flush staging into the snapshot engine, removing the Pebble dbcache layer (OpenWithCache and the whole dbcache package). FlatKV config switches from *CacheConfig to SnapshotEngineConfig (AccountStoreConfig, etc.) with flush and metrics knobs; Grafana queries rename pebblecache_* to snapshot_engine_*.

rootmulti now owns block height for the state-commit path: nextVersion() drives flush, working hash, and Commit, and flushedVersion skips repeat empty flushes when baseapp calls flush three times per block—fixing flatkv_only advancing height twice per block. SetInitialVersion sets lastCommitInfo.Version so seeded chains commit at the right height.

Snapshot lifecycle tests and behavior shift from SetHash/AwaitHash to Finalize/AwaitFlush; iterators take optional bounds and no longer block writes on shards. New flatkv_only height/regression tests, rollback-one-block coverage, and clearer migration-batch timing tests; config golden files updated.

^{Reviewed by Cursor Bugbot for commit 407bf3b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Aug 19, 2026, 3:53 PM

[yzang2019]

What does out of service mean for a shard? When would a shard become out of service?

[cody-littley]

Two scenarios qualify as "out of service":

• The shard is shutdown. Only reachable if you call shutdown concurrently with an operation that touches a shard (which is illegal).
• The shard has crashed. Only possible when the DB experiences a crash. DB failures are fatal, so we should never try to recover them.

Updated the doc to spell this out.

[blindchaser]

is looks RootHash commits the pending block. then later composite commit increments FlatKV again? is there a double commits on flatkv?

[cody-littley]

This is intentional on my part. The core problem is that comsos wants the hash before the commit, but the proper shape of the storage engine is to provide a hash only after the block is finalized+committed.

I've set it up so that if we call Commit() multiple times for the same block number, later commits become no-ops. If you try to write changes to the block after the commit, you get an error. Previously it was technically possible to write changes after we get the hash but before the commit, but this would either mean the hash is actually an invalid hash for that block.

Worth a discussion in parking lot.

[blindchaser]

Config.Fsync is not passed to the five snapshot-engine FlushSync fields. Since FlushSync defaults to false, programmatic callers that set Fsync=true still get unsynchronized normal flushes. can we propagate this value to all five store configurations and add a test.

[cody-littley]

good point, fixed

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0e3dee4. Configure here.}

[yzang2019]

FlatKV only refuses when the batch is stamped at the already-committed height N. The writer stamps N+1, so it silently defers instead. Someone reading this comment would conclude the non-empty case is caught downstream, and it isn't.

And the safety property currently rests on a convention — "nothing writes to the multistore in the preCommitHandler" — rather than an invariant. Whoever adds the next preCommitHandler hook, or a module that writes during it, gets silent height drift rather than an error. Given the PR's own stated principle about guarding at the choke point, turning flushedVersion == currentVersion && len(changeSets) > 0 into an error in flush() would make it checkable. That's a small, self-contained follow-up, not something that needs to hold up phase 1.

Might be worth adding a comments for context

[yzang2019]

This PR moves the FlatKV seal earlier: flatKVWorkingHash now calls flatKV.Commit(version) from inside GetWorkingHash, which runs in FinalizeBlock. memIAVL still commits later, in ABCI Commit. So for the whole gap between those two ABCI calls, FlatKV is at block N while memIAVL is at N-1.

The result is a CommitInfo labeled Version: N-1, carrying memIAVL's store hashes at N-1, with an evm_lattice entry holding block N's hash. That combination corresponds to no committed state at any height. Note the contrast with flatKVWorkingHash, which does compare hashed != version and panics — the height check exists exactly where the race can't happen, and is dropped where it can.

It's reachable. rootmulti.Query recomputes commit info live on the latest path, under no lock, and RPC nodes serve queries concurrently with block execution.
if latest { // latest never needs historical LoadVersion clone store = types.Queryable(commitment.NewStore(rs.scStore.GetChildStoreByName(storeName))) commitInfo = convertCommitInfo(rs.scStore.LastCommitInfo()) commitInfo = amendCommitInfo(commitInfo, rs.storesParams) }

The race technically predates the PR (memIAVL and FlatKV committed sequentially inside composite.Commit, so they disagreed for a few microseconds), but this widens it from an instruction window inside one function to the full inter-ABCI-call gap, on every block.