test(netsim): Docker NAT matrix with measured attach, TURN fallback and resume (P2) - #82
Conversation
…nd resume (P2) `make netsim` puts a real `mir up` agent and a real client behind separate simulated NATs in Docker, drives the production attach path between them, swaps the client's uplink mid-session, and writes a results table. Seven scenarios cover routable, port-restricted cone and symmetric NATs, with STUN, with TURN, and with every direct UDP path blocked so only TURN can work; two of them flip the client's network mid-session. The driver is a Go test binary built from a new test-only package, `go/internal/netsim`. Building it with `go test -c` is what lets a headless Linux container hold an owner identity: `internal/client` accepts the MIR_TEST_KEYCHAIN_DIR override only when argv[0] ends in ".test", so the production keychain rule stays intact. No production Go or JS changed. Measured on OrbStack (arm64), median of 3 reps: open-agent attach 223 ms prc-prc attach 822 ms sym-sym-stun fails, as it must — STUN cannot traverse two symmetric NATs sym-sym-turn attach 1245 ms turn-only attach 1262 ms flip-prc attach 823 ms, resume 4077 ms, continuation 3/3 flip-turn attach 1237 ms, resume 4496 ms, continuation 3/3 Resume splits into ~3.23 s to notice the dead link and ~0.83 s to redial, so the R1 gate of under 3 s needs earlier detection, not a faster redial. CI gets .github/workflows/netsim.yml on workflow_dispatch plus a weekly cron. It never gates a PR. Closes #71 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KeiotDVE94wEzvc7wcvm1y
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6d7a9d3393
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if res.Expect == "fail" { | ||
| if ok > 0 { | ||
| t.Logf("NOTE: %s was expected to fail but %d/%d reps succeeded — update the scenario's expectation", | ||
| res.Scenario, ok, len(res.Samples)) | ||
| } | ||
| return |
There was a problem hiding this comment.
When any sym-sym-stun repetition connects, this branch merely logs a note and exits successfully. That scenario is the matrix's negative control proving that peers cannot bypass the simulated symmetric NATs without TURN, so an unexpected connection can indicate leaked routing or TURN access and makes the other measurements untrustworthy; fail the scenario so the weekly workflow forces the topology or expectation to be investigated.
Useful? React with 👍 / 👎.
| if ok == 0 { | ||
| t.Fatalf("scenario %s: every rep failed", res.Scenario) | ||
| } |
There was a problem hiding this comment.
For every scenario expected to pass, the driver exits nonzero only when all repetitions fail. With the default three repetitions, a regression yielding one success and two failures therefore returns success, so run.sh never adds the scenario to FAILED and the scheduled workflow stays green despite most attaches or resumes failing; enforce the intended repetition failure threshold rather than accepting any single success.
Useful? React with 👍 / 👎.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6d7a9d3. Configure here.
| t.Logf("NOTE: %s was expected to fail but %d/%d reps succeeded — update the scenario's expectation", | ||
| res.Scenario, ok, len(res.Samples)) | ||
| } | ||
| return |
There was a problem hiding this comment.
Expected-fail success never fails
Medium Severity
When NETSIM_EXPECT=fail, any successful attach only logs a note and the test still passes. sym-sym-stun is the canary that the NATs are real; if a host-route leak returns and that scenario connects, the run stays green and the report still says it failed.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 6d7a9d3. Configure here.
| if ! wait_for 90 "the agent to register with the relay" agent_registered; then | ||
| docker compose logs --no-color "$AGENT_SVC" | tail -40 >&2 | ||
| return 1 | ||
| fi |
There was a problem hiding this comment.
Aborted run publishes stale results
Low Severity
If the relay or agent never becomes ready, run_scenario returns 1 and set -e aborts before the report step. results.md is never replaced, so the workflow’s if: always() summary reprints the last committed 18/18 table from a failed job.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 6d7a9d3. Configure here.
… gate (#84) netsim (#82) measured resume at 4.08s direct and 4.50s relayed, of which 3.23s was detection: iceDisconnectedTimeout (2s) plus LinkGrace (1s). The redial it gates was already sub-second, so the beta's "under 3s" could only come out of detection. iceDisconnectedTimeout 2s -> 1s (two missed 500ms keepalives instead of four) and LinkGrace 1s -> 500ms. Keepalive is unchanged, so a single lost packet still cannot tear down a live session. Measured over 9 flip reps each, medians: flip-prc resume 4077ms -> 2564ms (detection 3.23s -> 1.75s) flip-turn resume 4496ms -> 2996ms Detection reads ~250ms above the 1.5s arithmetic because pion checks liveness on a keepalive-driven ticker, so the transition lands up to one interval late. Flap accounting is untouched and stayed honest: every one of 18 flips redialed on the first attempt with no backoff sleep (redial 822-832ms direct, i.e. the healthy-drop path), 18/18 continuations held, and no session was torn down before its flip. Also here: - peer_link_test.go pins the detection-plus-redial sum against the 3s gate, so a future retune has to face it deliberately. - netsim records redials per rep and reports any that needed more than one; that is the signal to watch after a detection retune. - netsim/run.sh honours an exported NETSIM_REPS, which it previously clobbered. Closes #83 Claude-Session: https://claude.ai/code/session_01KeiotDVE94wEzvc7wcvm1y Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Netsim (#82, #84) measured the matrix; the gap that remains is real-network data, so say that instead. Claude-Session: https://claude.ai/code/session_01KeiotDVE94wEzvc7wcvm1y Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 participant
make netsimruns the whole NAT matrix on your laptop in under four minutes. Itputs a real
mir upagent and a real client behind separate simulated NATs inDocker, drives the production attach path between them, swaps the client's
uplink mid-session, and writes down what happened.
It measures the three numbers the v0.8 beta gates ask for: time to attach, time
to resume after a network flip, and whether the work survived the flip.
Closes #71.
Measured numbers
Run on OrbStack (arm64), 3 reps per scenario. Full table:
netsim/results/results.md.open-agentprc-prcsym-sym-stunsym-sym-turnturn-onlyflip-prcflip-turn18/18 reps connected across the scenarios that should connect. 6/6 flips kept
the session's work alive.
sym-sym-stunis excluded from those rates: it iswritten to fail, and does.
Two findings
Resume is ~4.1 s, and detection is nearly all of it. The flip splits into
~3.23 s to notice the link is dead, then ~0.83 s (direct) or ~1.23 s (TURN) to
carry bytes again. The 3.23 s is by design —
peer.iceDisconnectedTimeout(2 s)plus
peer.LinkGrace(1 s) — and the comment on those constants predicts "~3 s".The harness confirms the prediction, which also settles how to reach the R1 gate
of under 3 s: the redial is already sub-second, so the time has to come out of
detection.
Continuation held every time. The job started before the flip was still
running after it in 6/6 flips, on the direct path and on the relayed one.
Topology
Agent and client share no network. Each NAT router also drops forwarding to the
other LANs, because the Docker host has a route to every bridge it created —
without that drop a peer reaches the other's private address straight through
the host and never touches a NAT. The first version of this harness "passed"
prc-prcthat way, in 217 ms, while traversing nothing. Closing the leak turnedthe same scenario into a real hole-punch at 822 ms.
The client has two uplinks behind two different NATs. The standby is held
administratively down so it gathers no ICE candidate until the flip.
What each NAT approximates
noneprcSNAT --to-source <router public ip>for UDP with a matchingDNATback in, andFORWARDlimited to established conntrack flowssymMASQUERADE --random-fully, same inbound restrictionprcuses an explicitSNATrather thanMASQUERADEon purpose. Portpreservation, and therefore endpoint-independent mapping, is then true by
construction instead of left to the kernel's port-allocation heuristics — which
did not preserve ports on this host, and made
prc-prcfail 4/4 before thechange.
BLOCK_PEER_UDP=1additionally drops every forwarded UDP flow except the onesto and from coturn, so no direct path can exist at all. That is how
turn-onlyisolates the TURN fallback.
There is no full-cone mode. It was built, measured, and removed. Expressing
one means letting the
DNATaccept NEW inbound UDP, and those conntrack entriesthen collide with the SNAT mapping the node is about to create for the same
peer: the mapping moves, ICE's checks stop being symmetric, and every pair fails
(3/3).
open-agentcovers the always-reachable case instead.Design
The driver is a Go test binary. It is built with
go test -cfrom a newtest-only package,
go/internal/netsim. That is the point, not a shortcut:internal/clientkeeps the owner root in the OS keychain and accepts theMIR_TEST_KEYCHAIN_DIRoverride only whenargv[0]ends in.test. Buildingthe driver this way lets a headless Linux container hold an owner identity
without weakening the production storage rule.
No production code changed. The driver calls
client.Attach, races the reallocators, and runs
client.ReconnectLoopWithunder the production policy —including its 5 s
MinHealthy, which is why each rep holds the session for 7 sbefore flipping.
ReconnectNotify.OnResumedalready existed and its commentalready said it was "the number the NAT-matrix work (P2) reads"; this reads it.
On the other side, a real
mir upserves a real tmux session.Pairing is pre-provisioned, the attach path is not.
TestNetsimProvisionwrites the pairing outcome — owner pin, owner-signed registration authorization,
sealed registry record, pinned host key — straight into the shared state volume
instead of driving the interactive handshake. Everything after that is the real
path.
Continuation is measured, not assumed. After the first attach the driver
starts a counting heartbeat inside the tmux session. After the flip it requires a
heartbeat with a higher counter, which can only exist if the job ran right
through the outage.
The flip is a real flip.
netsim-flip.shbrings the standby uplink up, movesthe default route, and takes the old link down: new interface, new address, new
NAT mapping. It toggles, so consecutive reps flip back and forth.
How to run it
Needs a running Docker; OrbStack is what this was developed on. One image serves
every role (103 MB, built in ~15 s warm). Each scenario gets a clean stack, so
nothing carries over between them.
CI:
.github/workflows/netsim.yml,workflow_dispatchwith an optional scenariolist, plus a Monday cron. It publishes the table to the job summary and uploads
netsim/results/. It never gates a PR — this needs Docker networking,NET_ADMINand several minutes, and a UDP hole-punch that flakes must not blocka merge.
run.shexits non-zero only when a scenario that should connect didnot.
Caveats
Read the numbers as relative, not absolute. Every link here is a Docker bridge
with a round-trip time near 0.1 ms, so what the timings show is protocol work —
handshakes, gathering, timers — with the transport cost removed. Real networks
add their own latency on top. What the harness is good at is comparing paths and
catching the day one of them stops working.
Checks
cd go && go test ./...green,gofmt -l .empty,go vet ./...clean,cd web && npm test141/141,./scripts/verify-reproducible.shstillbyte-identical. The new package skips itself unless
MIR_NETSIM=1, sogo test ./...stays hermetic.🤖 Generated with Claude Code
https://claude.ai/code/session_01KeiotDVE94wEzvc7wcvm1y
Note
Low Risk
Test-only harness and CI/docs; no changes to production attach or relay binaries beyond what the existing client already exposes for
.testkeychain override.Overview
Adds a Docker-based NAT matrix (
make netsim/netsim/run.sh) that runs a realmir upagent and client behind simulated port-restricted and symmetric NATs, with optional coturn and strict “no TURN” scenarios. Each scenario gets a clean stack (relay, identities, iptables rules) so results are not contaminated across runs.The driver lives in
go/internal/netsimas ago test -cbinary that only runs whenMIR_NETSIM=1, so normalgo test ./...stays fast. It pre-provisions pairing state (instead of interactivemir pair), then exercises productionclient.AttachandReconnectLoopWith, recording attach time, resume after an uplink flip (netsim-flip.sh), and tmux continuation via heartbeat counters. Raw JSON lands innetsim/results/raw/;TestNetsimReportwritesnetsim/results/results.md(baseline numbers committed).Supporting pieces: shared
miranda-netsim:devimage, compose topology with internal LANs and NAT routers that block host bypass,.dockerignoreto shrink build context, README/Makefile docs, and.github/workflows/netsim.yml(weekly cron + manual dispatch, not a PR gate) publishing the results table and artifacts.Reviewed by Cursor Bugbot for commit 6d7a9d3. Bugbot is set up for automated code reviews on this repo. Configure here.