Skip to content

v0.8.3: bound admission memory; ack only after hand-off (#157) - #158

Merged
dendisuhubdy merged 1 commit into
mainfrom
fix/v083-admission-memory
Oct 28, 2026
Merged

dendisuhubdy merged 1 commit into
mainfrom
fix/v083-admission-memory

Conversation

@dendisuhubdy

Copy link
Copy Markdown
Member

Closes #157

Design

One byte budget (admission::AdmissionBudget, a tokio semaphore counted in bytes, shared by both RPC listeners and the node loop). It is never waited on: a charge that does not fit is refused at once.

holder charge released
RPC POST body (rpc::charge_body middleware, runs before axum reads the body) Content-Length, capped at the body limit; the whole limit if none when the response is built. rand_sendTransaction waits for its verdict inside this window, so the hex body stays charged
verify queue, gossip and RPC (Node::on_gossiped_tx, Node::submit_tx) admission::charge_of(tx) = bincode::serialized_size, no encode when Verdict (it carries the charge) is dropped at the end of on_verdict

When the budget is full:

  • RPC answers -32005 (busy, retry), HTTP 503 with Retry-After: 1, and data: {wanted, available, limit}. The refused body is drained chunk by chunk and never buffered. Without the drain, hyper closes the connection on a client that is still writing, and the client sees a reset instead of the error. A refusal in the node loop (the verify-queue charge) is the new MempoolError::AdmissionBusy. It is retryable and never cached, and the RPC maps it to the same -32005. rand already backs off and retries -32005 (is_rate_limited, 5 tries over about 31 s), so the CLI needs no change.
  • Gossip reports Ignore, never Reject, so the forwarder isn't penalised. Nothing goes in the refused cache, and a park waiting for the body is still filled.

Default budget: 1/16 of total RAM, clamped to 64–256 MiB, and never under one largest submission (body cap + block cap). That gives about 117 MiB on the fleet's 1967 MB droplets and 256 MiB on hosts with about 4 GB or more. --admission-budget-mb overrides it. A value under the floor is refused at startup. RAM is read with randprotocol_prover::memory::total_bytes() (sysinfo, already a dependency). It is wired through RpcOptions.admission_budget, so the many literal NodeConfigs stay untouched. rand_status gains admission_bytes and admission_budget_bytes.

Ack semantics: found, documented, not changed

The code already acknowledged only after hand-off. The RPC reply is sent from Node::on_verdict only after the transaction verified, insert_verified pooled it at the current tip, and net.broadcast handed it to the swarm task. No path returns a hash for a transaction that is only queued. This PR documents that in docs/rpc.md ("What a hash means"). The gap left is the in-process window between handing the transaction to the swarm task and the first peer receiving it. The 18 lost acknowledgements in the issue fell in that window when the OOM killer ran, and the budget is what prevents those kills.

Ops

deploy/vps-setup.sh and deploy/cutover-droplet-rand.sh units set MemoryHigh=85%, which makes the kernel reclaim and throttle instead of OOM-killing. There is deliberately no MemoryMax. docs/node-hardware.md now gives 4 GB as the validator minimum and keeps the swap advice. Nothing was deployed.

Tests (deterministic: every step waits on a message or a return, never on a clock)

  • admission::tests::the_admission_budget_refuses_past_its_limit_and_releases_on_drop: fills to exactly the limit and refuses one byte more. Bytes come back on drop, a clone shares the budget, a charge larger than the whole budget is always refused, and a zero charge always fits.
  • admission::tests::the_default_admission_budget_follows_the_host and a_transaction_is_charged_its_encoded_length.
  • node::tests::the_verify_queue_is_bounded_by_the_admission_budget: with no verify slots, gossip past the budget is reported Ignore and not cached, and RPC past it gets AdmissionBusy (retryable). Handling the verdict releases the charge, and the next transaction is then queued.
  • rpc::tests::concurrent_large_submissions_past_the_admission_budget_are_told_to_retry: 8 concurrent ~1.6 MB rand_sendTransactions over real HTTP against a budget of 3 bodies, with a stand-in node loop that holds its replies. Exactly 3 reach the node and 5 get 503 / -32005 / Retry-After. in_use is exactly 3 bodies, and no 4th submission arrives. After release, all 3 succeed, in_use returns to 0, and a retry goes through. It passed 30 of 30 repeated runs.

Local runs (debug): node lib 774 passed. The 1 failure is msauth_key::an_exe_in_the_tree_target…, which is environmental: it needs the test binary under this tree's target/, and I used a shared CARGO_TARGET_DIR. Integration tests: network 14, rpc_contract 11, submit 2, ws 9. Prover lib 14. cargo clippy --no-deps -p randprotocol-node -p randprotocol-prover --all-targets -D warnings is clean. cargo fmt was not applied: the repo isn't rustfmt-formatted and CI doesn't check it, and running it rewrites about 40k lines.

Merge notes for the other v0.8.3 PRs

🤖 Generated with Claude Code

https://claude.ai/code/session_01GFs5h141Ct6RTGGrCTQ1vy

One byte budget (AdmissionBudget, a tokio semaphore counted in bytes) over
the memory admission holds before a transaction is pooled:

- RPC: every POST body is charged its Content-Length (else the body cap)
  before axum reads it, held until the response is built, so a
  rand_sendTransaction stays charged while it waits for its verdict. A
  charge that does not fit is answered at once with -32005 (HTTP 503,
  Retry-After: 1); the body is drained chunk by chunk, never buffered, so
  the client reads the error instead of a reset.
- Verify queue: each queued transaction (gossip or RPC) carries a charge of
  its bincode length, released when its verdict has been handled. Gossip
  past the budget is reported Ignore (never Reject); RPC gets
  MempoolError::AdmissionBusy, answered -32005.

Default: 1/16 of the host's RAM within 64-256 MiB (about 117 MiB on the
fleet's 2 GB droplets), never under one largest submission (body cap plus
block cap). --admission-budget-mb overrides; a value under that floor is
refused at startup. rand_status gains admission_bytes and
admission_budget_bytes.

Ack semantics were already "verified, pooled and handed to gossip" (the
RPC reply is sent from Node::on_verdict after insert_verified and
broadcast); now documented in docs/rpc.md. The unit templates set
MemoryHigh=85%, and docs/node-hardware.md makes 4 GB the validator minimum.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GFs5h141Ct6RTGGrCTQ1vy
@codecov

codecov Bot commented Oct 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@dendisuhubdy
dendisuhubdy merged commit 9095436 into main Oct 28, 2026
23 checks passed
@dendisuhubdy
dendisuhubdy deleted the fix/v083-admission-memory branch October 28, 2026 01:25
dendisuhubdy added a commit that referenced this pull request Oct 28, 2026
#158 (issue #157) makes every transaction on the verify queue carry an
AdmissionCharge against the node's one byte budget, given back when its
Verdict is dropped at the end of Node::on_verdict. #155 (issue #145)
added a third producer for that queue, Node::on_stem_tx, which still
pushed (tx, VerifySource::Stem(from)) without a charge: the merged tree
did not compile (E0308 at the push, E0063 at six test Verdict literals).

- on_stem_tx takes admission::charge_of(&tx) from the budget after the
  existing pre-screens (stem allowance, refused cache, queue bound,
  pooled, cached proof refusal, precheck) and before the queue push, so
  a stemmed transaction is charged exactly like a gossiped or submitted
  one and released on its verdict — not when its stem ends; the
  stempool holds a verified transaction, not a queued one.
- Past the budget the stem is dropped the way gossip sheds one: a debug
  line, nothing cached in the refused cache, nothing reported or sent,
  the peer not penalised. The predecessor's embargo fluffs it if no
  other copy gets through, as with any other stem drop.
- The six #155 node tests that build a Verdict by hand pass
  AdmissionCharge::none(); new test
  a_stemmed_transaction_is_charged_and_shed_unpenalised_past_the_budget.
- docs/rpc.md "What a hash means": the hash means verified and then
  stemmed or broadcast (held in the stempool and handed to the stem
  successor, or pooled and handed to gossip), not "handed to gossip";
  the -32005 section says a stem is charged and shed alike. The
  rand_sendTransaction comment in rpc.rs and the #157 CHANGELOG entry
  say the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GFs5h141Ct6RTGGrCTQ1vy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Validators OOM-killed under bursts of large transactions; acknowledged transactions lost

1 participant