Conversation
jsonParsed is the heaviest getBlock encoding (~35-40% larger payload and
higher server-side cost). Benchmarks against public mainnet RPC show json
cuts both latency and bandwidth substantially, improving real-time indexing
throughput so Solana keeps up with slot production instead of falling behind
into catchup.
Switching encoding requires two adjustments the RPC used to do for us:
- AccountKey now unmarshals from both a bare pubkey string (json) and the
{ pubkey, signer, writable } object (jsonParsed).
- Versioned (v0) transactions deliver Address Lookup Table accounts in
meta.loadedAddresses rather than merged into message.accountKeys. Index
resolution now appends them as static keys + loaded writable + loaded
readonly, matching the order the RPC produces under jsonParsed.
Solana produces skipped slots that will never have a block. The catchup and rescanner paths already recognise these, but the real-time regular path ran them through handleBlockResult, which persisted each skipped slot as a failed block and pushed it to the rescanner — costing an extra getBlock per skip just to confirm the skip, exactly when block production is fastest. Detect ErrorTypeBlockNotFound on Solana in the non-reorg batch loop, advance currentBlock past the slot, and notify the observer as not-found instead of failed.
The failover only penalised latency on the error path (analyzeError's >3s check). A provider that returns successfully but slowly — an overloaded free RPC responding in seconds without ever erroring — was never rotated away, so it kept dragging down real-time throughput. Add a success-path latency check in executeCore: a call slower than SlowResponseThreshold blacklists the provider for SlowResponseCooldown so the next call prefers a faster one. It never drops the available pool below MinActiveProviders, so when every provider is slow we keep using them. Both thresholds are config fields (defaulting to the previous hard-coded 3s / 2m), and analyzeError now reads the same values.
The defaults.failover yaml block was dead configuration: the Failover field lived only on the Defaults struct, was never merged into ChainConfig, and every NewFailover call passed nil — so rpc.DefaultFailoverConfig() was always the effective source. Remove the unused field and the yaml block so failover tuning lives in one place (code) instead of misleading knobs operators can set with no effect.
Replace the static per-batch semaphore in GetBlocksByNumbers with a shared AIMD concurrency limiter. The configured throttle.concurrency becomes a ceiling: the limiter multiplicatively backs off when getBlock calls get slow (>2.5s) or fail, and additively recovers toward the ceiling once calls settle (<1.2s). It never exceeds the ceiling, so it can only match or beat the previous static behaviour. Because the Solana indexer is built once and shared across all worker modes, the limiter is a single global congestion controller for the chain's getBlock load — replacing the previous uncoordinated per-call semaphores (regular + catchup + rescanner each had their own). This adapts call pressure to real RPC capacity instead of a fixed number, avoiding the rate-limit death spiral on overloaded free nodes while still using full concurrency when they are healthy. New pkg/adaptive holds the reusable limiter with full unit + race coverage.
Some free RPCs (e.g. drpc) reject a chain outright with 'not available on free plan' / code 35. That was classified as a generic error, so the provider only degraded after ForceRotateThreshold wasted attempts every pass. Classify it as chain_unavailable and blacklist for 24h, since the condition is permanent for that node.
…sted When every provider is rate-limited/blacklisted, many concurrent callers each triggered emergency recovery, un-blacklisting a provider only to have it 429 again immediately — a hot recover->fail->recover loop that spammed logs and hammered dead nodes. Space emergency recoveries by EmergencyRecoveryInterval (default 2s): within the window, callers get errAllProvidersBackoff and retry instead of churning. Also refresh the example Solana config: drop drpc (does not serve Solana free), add keyed-node placeholders (Helius/Ankr/Chainstack) since free pools cannot sustain getBlock at slot rate.
The worker logger already binds chain (and mode) via logger.With, so passing "chain" again in each call printed it twice (chain=X chain=X). Remove the redundant explicit chain args; the value now comes only from logger context.
Bloom init/sync iterate all network types, but the DB's address_type enum may lack some (e.g. 'apt'), so filtering wallet_addresses by that type fails with 22P02 and aborts init. Treat 22P02 (invalid enum value) as no matching rows in both the repository (WrapError) and the raw loader query, so unknown types are simply skipped.
This was referenced Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Base of the stack. Solana getBlock performance/reliability and RPC failover improvements.
jsonencoding for getBlock instead ofjsonParsed(cheaper)Stack: this PR → refactor/kvstore-redis → fix/catchup-redis-hash