consensus-lab

consensus-lab is a deterministic, event-sourced Rust workspace for studying distributed consensus under realistic but fully simulated network conditions. It is designed as a peer project to exchange-lab: the same emphasis on replayability, journal-first state, seeded fault injection, snapshots, metrics, and tooling, but aimed at Raft and consensus safety instead of market microstructure.

Goals

• Deterministic, reproducible simulation of Raft-like consensus protocols
• Event-sourced state transitions with append-only journals and snapshot checkpoints
• Seed-driven fault modeling for partitions, loss, delay, duplication, reordering, and Byzantine tampering
• Operational ergonomics through a CLI, gRPC/REST control plane, and Prometheus metrics
• Testing hooks for determinism, property-based workloads, and fault scenario regression coverage

Workspace Layout

consensus-lab/
|-- Cargo.toml
|-- README.md
|-- data/
|   |-- topologies/
|   `-- workloads/
|-- scripts/
`-- crates/
    |-- core/      # shared domain types, topology, events, snapshots, faults
    |-- raft/      # deterministic Raft state machine
    |-- sim/       # discrete-event engine and network/fault model
    |-- journal/   # append-only log, CRC, sparse index, snapshot manifests
    |-- replay/    # deterministic replay/reporting
    |-- workload/  # generators and linearizability checks
    |-- metrics/   # Prometheus instrumentation
    |-- api/       # tonic gRPC + axum REST control plane
    `-- cli/       # clap-based operational interface

Architecture

                +---------------------+
                |  consensus-lab-cli  |
                | generate / run /    |
                | replay / verify     |
                +----------+----------+
                           |
                           v
 +-------------+   +-------+--------+   +------------------+
 | workload    |-->| sim engine     |-->| journal          |
 | generators  |   | virtual time   |   | cluster + node   |
 | linearizab. |   | BinaryHeap<Q>  |   | CRC + sparse idx |
 +-------------+   +-------+--------+   +--------+---------+
                           |                         |
                           v                         v
                   +-------+--------+       +--------+---------+
                   | raft logic     |       | replay engine    |
                   | pure step fn   |       | deterministic    |
                   | ordered inputs  |       | resume / compare |
                   +-------+--------+       +--------+---------+
                           |
                           v
                   +-------+--------+
                   | api + metrics  |
                   | tonic + axum   |
                   | prometheus     |
                   +----------------+

Determinism Model

• All non-determinism is routed through StdRng::seed_from_u64(seed).
• Virtual time is explicit (SimTime), and event dispatch is ordered by (time, sequence) in a BinaryHeap.
• Fault decisions are made during message dispatch, recorded as immutable ClusterEvents, and can be replayed with the same seed.
• verify-determinism runs identical simulations twice and compares serialized event logs plus final node snapshots.

Fault Model

The current scaffold exposes deterministic control points for:

• Packet loss (packet_loss_pct)
• Delay jitter (baseline_latency_ms plus log-normal or Pareto jitter)
• Message duplication
• Message reordering (reorder_window)
• Scheduled partitions (PartitionRule)
• Simplified Byzantine behavior
  • Payload tampering
  • Selective target drops
  • Sender forgery is represented in the proto/API shape and is ready for expansion
• Leader crash timing hooks (leader_crash_at exists in FaultSpec for future orchestration)

This repository intentionally keeps the simulation offline: there is no real socket I/O between nodes. All transport is modeled through scheduled SimEvent::DeliverRpc events.

Key Components

core

• Strongly typed identifiers (NodeId, Term, LogIndex)
• Raft RPC types (AppendEntries, RequestVote, responses)
• Immutable cluster event schema
• Topology and fault definitions
• Snapshot structures for node persistence

raft

• Deterministic RaftNode
• Tick-driven elections and heartbeats
• Simplified log append / commit advancement
• Snapshot export of persistent + volatile state

sim

• Simulator orchestrates nodes, event queue, virtual time, and fault injection
• DefaultFaultInjector models loss, partitions, duplication, jitter, and simple tampering
• SimulationSummary captures leader history and final snapshots

journal

• Append-only cluster and node journals
• CRC32 validation per frame
• Sparse index structures for faster seek/checkpoint workflows
• Snapshot manifests carrying the journal cursor required for resume

replay

• ReplayEngine loads a cluster journal and regenerates a deterministic summary
• Replay fingerprints can be used for CI regression checks and golden traces

workload

• Seeded workload generators
• Linearizability sanity checking over committed client sequences
• Property-based tests for generated sequence invariants

api

• ControlService gRPC definition in crates/api/proto/consensuslab/v1/sim.proto
• Streaming event access, node state queries, leader history, fault injection, pace control
• REST endpoints:
  • GET /healthz
  • GET /metrics
  • GET /v1/nodes/:node_id
  • GET /v1/leaders

metrics

• Prometheus histograms for message latency, election timeout, commit latency, throughput
• Counter family for fault observations

CLI

The main binary is consensus-lab-cli.

cargo run -p consensus-lab-cli -- generate-topology --nodes 5
cargo run -p consensus-lab-cli -- ingest-workload --ops 1000 --seed 42
cargo run -p consensus-lab-cli -- run-simulation --mode max-speed
cargo run -p consensus-lab-cli -- replay --with-faults --seed 42 --packet-loss-pct 2.5
cargo run -p consensus-lab-cli -- verify-determinism --seed 42
cargo run -p consensus-lab-cli -- snapshot
cargo run -p consensus-lab-cli -- resume --snapshot node-1.snap
cargo run -p consensus-lab-cli -- bench-throughput --iterations 500
cargo run -p consensus-lab-cli -- bench-latency --iterations 500
cargo run -p consensus-lab-cli -- serve --grpc-addr 127.0.0.1:50051 --rest-addr 127.0.0.1:8080

Example Data

• [data/topologies/three-node.json](/mnt/c/Users/joann/Desktop/Github Shart/data/topologies/three-node.json)
• [data/workloads/counter-burst.json](/mnt/c/Users/joann/Desktop/Github Shart/data/workloads/counter-burst.json)

These files are intentionally small and human-readable so they can act as canonical smoke-test fixtures.

Build and Test

cargo check
cargo test
cargo run -p consensus-lab-cli -- verify-determinism --seed 7

Demo Scripts

• POSIX: [scripts/demo-local.sh](/mnt/c/Users/joann/Desktop/Github Shart/scripts/demo-local.sh)
• Windows CMD: [scripts/demo-local.bat](/mnt/c/Users/joann/Desktop/Github Shart/scripts/demo-local.bat)
• PowerShell: [scripts/demo-local.ps1](/mnt/c/Users/joann/Desktop/Github Shart/scripts/demo-local.ps1)

Extension Roadmap

This scaffold is intentionally structured so you can deepen it without reworking the architecture:

1. Expand Raft replication bookkeeping (next_index, match_index) into full safety and quorum commit semantics.
2. Persist binary journal frames instead of JSON lines while keeping the same CRC/index model.
3. Add richer Byzantine scenarios (forged sender IDs, stale-term spoofing, equivocation).
4. Promote the replay engine from summary reconstruction to exact message-by-message re-execution from recorded cluster traces.
5. Add scenario suites that mirror Jepsen-style nemesis schedules and publish exported metrics per run.