Repository navigation
v0.8.3: bound admission memory; ack only after hand-off (#157) - #158
Merged
Merged
Conversation
One byte budget (AdmissionBudget, a tokio semaphore counted in bytes) over the memory admission holds before a transaction is pooled: - RPC: every POST body is charged its Content-Length (else the body cap) before axum reads it, held until the response is built, so a rand_sendTransaction stays charged while it waits for its verdict. A charge that does not fit is answered at once with -32005 (HTTP 503, Retry-After: 1); the body is drained chunk by chunk, never buffered, so the client reads the error instead of a reset. - Verify queue: each queued transaction (gossip or RPC) carries a charge of its bincode length, released when its verdict has been handled. Gossip past the budget is reported Ignore (never Reject); RPC gets MempoolError::AdmissionBusy, answered -32005. Default: 1/16 of the host's RAM within 64-256 MiB (about 117 MiB on the fleet's 2 GB droplets), never under one largest submission (body cap plus block cap). --admission-budget-mb overrides; a value under that floor is refused at startup. rand_status gains admission_bytes and admission_budget_bytes. Ack semantics were already "verified, pooled and handed to gossip" (the RPC reply is sent from Node::on_verdict after insert_verified and broadcast); now documented in docs/rpc.md. The unit templates set MemoryHigh=85%, and docs/node-hardware.md makes 4 GB the validator minimum. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GFs5h141Ct6RTGGrCTQ1vy
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
dendisuhubdy
added a commit
that referenced
this pull request
Oct 28, 2026
#158 (issue #157) makes every transaction on the verify queue carry an AdmissionCharge against the node's one byte budget, given back when its Verdict is dropped at the end of Node::on_verdict. #155 (issue #145) added a third producer for that queue, Node::on_stem_tx, which still pushed (tx, VerifySource::Stem(from)) without a charge: the merged tree did not compile (E0308 at the push, E0063 at six test Verdict literals). - on_stem_tx takes admission::charge_of(&tx) from the budget after the existing pre-screens (stem allowance, refused cache, queue bound, pooled, cached proof refusal, precheck) and before the queue push, so a stemmed transaction is charged exactly like a gossiped or submitted one and released on its verdict — not when its stem ends; the stempool holds a verified transaction, not a queued one. - Past the budget the stem is dropped the way gossip sheds one: a debug line, nothing cached in the refused cache, nothing reported or sent, the peer not penalised. The predecessor's embargo fluffs it if no other copy gets through, as with any other stem drop. - The six #155 node tests that build a Verdict by hand pass AdmissionCharge::none(); new test a_stemmed_transaction_is_charged_and_shed_unpenalised_past_the_budget. - docs/rpc.md "What a hash means": the hash means verified and then stemmed or broadcast (held in the stempool and handed to the stem successor, or pooled and handed to gossip), not "handed to gossip"; the -32005 section says a stem is charged and shed alike. The rand_sendTransaction comment in rpc.rs and the #157 CHANGELOG entry say the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GFs5h141Ct6RTGGrCTQ1vy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #157
Design
One byte budget (
admission::AdmissionBudget, a tokio semaphore counted in bytes, shared by both RPC listeners and the node loop). It is never waited on: a charge that does not fit is refused at once.rpc::charge_bodymiddleware, runs before axum reads the body)Content-Length, capped at the body limit; the whole limit if nonerand_sendTransactionwaits for its verdict inside this window, so the hex body stays chargedNode::on_gossiped_tx,Node::submit_tx)admission::charge_of(tx)=bincode::serialized_size, no encodeVerdict(it carries the charge) is dropped at the end ofon_verdictWhen the budget is full:
-32005(busy, retry), HTTP 503 withRetry-After: 1, anddata: {wanted, available, limit}. The refused body is drained chunk by chunk and never buffered. Without the drain, hyper closes the connection on a client that is still writing, and the client sees a reset instead of the error. A refusal in the node loop (the verify-queue charge) is the newMempoolError::AdmissionBusy. It is retryable and never cached, and the RPC maps it to the same-32005.randalready backs off and retries-32005(is_rate_limited, 5 tries over about 31 s), so the CLI needs no change.Ignore, neverReject, so the forwarder isn't penalised. Nothing goes in the refused cache, and a park waiting for the body is still filled.Default budget: 1/16 of total RAM, clamped to 64–256 MiB, and never under one largest submission (body cap + block cap). That gives about 117 MiB on the fleet's 1967 MB droplets and 256 MiB on hosts with about 4 GB or more.
--admission-budget-mboverrides it. A value under the floor is refused at startup. RAM is read withrandprotocol_prover::memory::total_bytes()(sysinfo, already a dependency). It is wired throughRpcOptions.admission_budget, so the many literalNodeConfigs stay untouched.rand_statusgainsadmission_bytesandadmission_budget_bytes.Ack semantics: found, documented, not changed
The code already acknowledged only after hand-off. The RPC reply is sent from
Node::on_verdictonly after the transaction verified,insert_verifiedpooled it at the current tip, andnet.broadcasthanded it to the swarm task. No path returns a hash for a transaction that is only queued. This PR documents that indocs/rpc.md("What a hash means"). The gap left is the in-process window between handing the transaction to the swarm task and the first peer receiving it. The 18 lost acknowledgements in the issue fell in that window when the OOM killer ran, and the budget is what prevents those kills.Ops
deploy/vps-setup.shanddeploy/cutover-droplet-rand.shunits setMemoryHigh=85%, which makes the kernel reclaim and throttle instead of OOM-killing. There is deliberately noMemoryMax.docs/node-hardware.mdnow gives 4 GB as the validator minimum and keeps the swap advice. Nothing was deployed.Tests (deterministic: every step waits on a message or a return, never on a clock)
admission::tests::the_admission_budget_refuses_past_its_limit_and_releases_on_drop: fills to exactly the limit and refuses one byte more. Bytes come back on drop, a clone shares the budget, a charge larger than the whole budget is always refused, and a zero charge always fits.admission::tests::the_default_admission_budget_follows_the_hostanda_transaction_is_charged_its_encoded_length.node::tests::the_verify_queue_is_bounded_by_the_admission_budget: with no verify slots, gossip past the budget is reportedIgnoreand not cached, and RPC past it getsAdmissionBusy(retryable). Handling the verdict releases the charge, and the next transaction is then queued.rpc::tests::concurrent_large_submissions_past_the_admission_budget_are_told_to_retry: 8 concurrent ~1.6 MBrand_sendTransactions over real HTTP against a budget of 3 bodies, with a stand-in node loop that holds its replies. Exactly 3 reach the node and 5 get 503 /-32005/Retry-After.in_useis exactly 3 bodies, and no 4th submission arrives. After release, all 3 succeed,in_usereturns to 0, and a retry goes through. It passed 30 of 30 repeated runs.Local runs (debug): node lib 774 passed. The 1 failure is
msauth_key::an_exe_in_the_tree_target…, which is environmental: it needs the test binary under this tree'starget/, and I used a sharedCARGO_TARGET_DIR. Integration tests: network 14, rpc_contract 11, submit 2, ws 9. Prover lib 14.cargo clippy --no-deps -p randprotocol-node -p randprotocol-prover --all-targets -D warningsis clean.cargo fmtwas not applied: the repo isn't rustfmt-formatted and CI doesn't check it, and running it rewrites about 40k lines.Merge notes for the other v0.8.3 PRs
(Transaction, VerifySource, AdmissionCharge). v0.8.3: Dandelion++ stem relay for transactions (#145) #155's stem push site will fail to compile until it takes a charge:self.admission.try_charge(admission::charge_of(&tx)), shedding onErr. That is intended, because stem transactions are admission memory too. Under v0.8.3: Dandelion++ stem relay for transactions (#145) #155 the RPC hash means "stemmed or fluffed" rather than "broadcast". The doc paragraph should say so at merge.decode_hex_bodyline inrand_sendTransactionis untouched here. This PR only changes the reply match below it.CHANGELOG.md"Unreleased" will conflict trivially with v0.8.3: node fixes (#143 #144 #146 #149) #150/v0.8.3: Dandelion++ stem relay for transactions (#145) #155/v0.8.3: prover/verify performance + faucet flag (#151 #152 #153) #156, which all replace "Nothing yet."🤖 Generated with Claude Code
https://claude.ai/code/session_01GFs5h141Ct6RTGGrCTQ1vy