Skip to content

testing-and-quality.md

Latest commit

 

History

History
127 lines (90 loc) · 10.7 KB

testing-and-quality.md

File metadata and controls

127 lines (90 loc) · 10.7 KB
Raw

Testing & quality

Correctness is a first-class feature of xtcp2, not an afterthought. Parsing raw kernel netlink bytes is unforgiving — a single wrong offset silently corrupts every record — so xtcp2 is validated against a large corpus of real captured netlink traffic (.pcap files) spanning many Linux kernel versions, decoded by explicit typed deserializers (no reflection), and backed by roughly 800 tests at over 92% statement coverage. This is one of the biggest improvements over the original xtcp: faster because the hot path avoids reflection, and far safer because the parsers are exercised against genuine kernel output.

Table of contents

Why this matters

The kernel's inet_diag reply is a packed sequence of C structs and typed attributes whose layout varies across kernel versions and architectures. Reading it correctly means matching the kernel's byte layout exactly. The original xtcp leaned on reflection-based decoding, which is slower and harder to verify. xtcp2 instead hand-builds typed deserializers and proves them correct against captured kernel traffic, so layout drift between kernel versions is caught by tests rather than discovered in production.

Captured netlink fixtures

The heart of the test suite is a corpus of 61 .pcap capture files under pkg/xtcpnl/testdata/, organized into per-kernel-version directories:

Kernel version directory Notes
4_19_319 Linux 4.19 LTS
5_4_281 Linux 5.4 LTS
5_15_164 Linux 5.15 LTS
6_1_103 Linux 6.1 LTS
6_6_44 Linux 6.6 LTS — the richest set (congestion variants, v4/v6, scale captures)
6_8_12 Linux 6.8
6_10_3 Linux 6.10 — includes long-running netem captures
7_0_3 newest captures

The fixtures cover a wide range of real situations:

  • Request / reply / dump-done exchanges — the full sock_diag conversation, captured single-packet so individual message parsing can be asserted byte-for-byte.
  • Scale — captures at 10, 100, 1000, 2000, and 10000 sockets, so batching and multi-packet dump handling are tested under realistic load.
  • Congestion-control variants — dedicated captures for BBR, DCTCP, and Vegas, exercising the algorithm-specific attribute deserializers.
  • IPv4 and IPv6 — both address families.
  • Long-running / netem captures — multi-minute captures (~30 and ~60 minutes) of 2000 sockets under simulated network impairment, capturing the kind of evolving tcp_info state a synthetic test could never produce.

Building this corpus was a significant effort: each capture is real kernel output recorded on the named kernel version, which is what makes the parser tests trustworthy.

Deserialization tests

The pkg/xtcpnl package decodes the fixtures and asserts the results. Representative test files:

  • pkg/xtcpnl/testdata_test.go — fixture loading and the path constants for the capture files.
  • pkg/xtcpnl/xtcpnl_inet_diag_msg_test.go, xtcpnl_inet_diag_msg_sockid_test.go, xtcpnl_inet_diag_reqv2_test.go — the inet_diag message header, socket ID, and request structures.
  • pkg/xtcpnl/xtcpnl_RTAttr_test.go, xtcpnl_nl_msg_hdr_test.go — netlink attribute and message-header parsing.
  • pkg/xtcpnl/xtcpnl_extract_7_0_3_fixtures_test.go — extracting and asserting against the newest fixtures.
  • pkg/xtcp/deserialize_corner_cases_test.go — corner cases in the daemon-side deserialize path.

Struct-size and field-offset assertions guard against silent layout regressions, and the golden proto-deserialization test (nix build .#test-proto-deserialize-golden) checks that decoding known-good fixtures still produces the expected records.

No reflection: faster and safer

xtcp2 parses each inet_diag attribute with an explicit, statically-typed decoder in pkg/xtcpnl/xtcpnl_inet_diag_*.go, dispatched through a typed deserializer registry (pkg/xtcp/deserializers.go). The hot collection path therefore does no reflection-based decoding — it reads fixed offsets directly into typed fields. Compared to a reflection-driven approach this removes per-field reflection overhead on the busiest code path, and because every decoder is covered by the fixture tests above, the speedup does not come at the cost of correctness.

pkg/xtcpnl contains no binary.Read call outside _test.go. What reflection remains is a set of binary.Read twins in xtcpnl_reflection_twins_test.go, kept deliberately as the control group that measures what the hand-written decoders buy. Every file holding one opens with a banner saying so — the reflection code is for performance comparison only and is strongly not recommended in production.

The numbers are not left to assertion. struct tcp_info is the widest decoder in the package, and on a real capture:

decoder ns/op B/op allocs/op
DeserializeTCPInfo, 248-byte kernel 6.10 layout 28.51 0 0
the same struct via binary.Read 2353 336 2
DeserializeTCPInfo, 280-byte kernel 7.0 layout with the Accurate ECN trailer 33.61 0 0
the same struct via binary.Read 2517 336 2

About 75× faster and allocation-free, and the 32 extra bytes of the 7.0 trailer cost roughly 5 ns. pkg/xtcpnl/xtcpnl_perf_gate_test.go turns that into a gate: 18 rows asserting 0 allocs/op on every manual decoder plus a minimum speedup over its twin, so the gate fails if reflection ever measures anywhere near a manual decoder — which would indicate a problem with the manual decoder, not a license to use reflection.

The speedup half is skipped under -race, and the reason is worth understanding rather than working around. The race detector instruments every memory access, so it taxes a manual decoder's ~70 individual field writes far more heavily, proportionally, than it taxes binary.Read's already-slow reflect work — it does not scale the two halves together the way host load does. The 280-byte TCPInfo ratio falls from 75× to 4.5× purely from turning the detector on, and across the whole table the ratios compress from 16.7×–336× down to 5.2×–44.9×. A ratio floor is therefore not measurable under -race, so test-go-race asserts only 0 allocs/op — host-independent, and it holds exactly under the detector — while test-go-unit asserts both halves.

Read the full -benchmem output with nix build .#test-go-bench && ./result/bin/xtcp2-go-bench, and pass -count=1 when timing by hand: go test caches results, and a cached run returns byte-identical timings that look like excellent stability and mean nothing.

Test coverage

The suite is large and the bar is high:

  • ~800 test functions across 107 _test.go files.
  • 92.4% overall statement coverage, with a 90%-per-package target — every package is green (roughly 90–96%). See the per-package table in quality-report.md.
  • A coverage baseline is tracked in docs/coverage-baseline.txt so regressions are caught.
  • Coverage from ordinary host test runs and from the microVM integration runs is merged (nix run .#coverage-merge) for a complete picture, including code paths — like setns and io_uring — that only execute inside a real kernel.

Benchmarks and fuzzing

  • 126 benchmark functions (e.g. pkg/xtcpnl/xtcpnl_bench_test.go) measure parsing throughput against the fixture corpus, so performance changes are observable.
  • Fuzz testing exercises the parser against malformed input.
  • Namespace-discovery A/B (tools/discovery-bench/) compares directory-scan vs /proc-inode-scan discovery. A hermetic microbenchmark runs under go test; the root-only real-kernel grid runs via the microvm-x86_64-discovery-bench flavor. See integration-testing.md.

Audit tools

Beyond unit tests, custom static-analysis tools under tools/ enforce project-specific invariants and run as part of nix flake check:

Tool / check Guards
netlink-audit Netlink parsing invariants.
iouring-audit The io_uring code path.
metrics-audit Prometheus metric registration.
proto-field-audit Protobuf field numbering / schema consistency.
proto-audit-netlink The netlink layout oracle: are pkg/xtcpnl's Go structs the shape the kernel actually sends? Compares against the Linux UAPI headers by wire bit offset rather than by field name, which is what lets it report a field the struct does not have at all — that is how the 11 missing Accurate ECN fields were found. Gating per protocol, via gatedProtocols in nix/checks/default.nix — NL_Diag_TCPInfo today, the only one whose deltas are fully triaged. The other 25 are advisory: their deltas are printed and written to $out/unallowlisted.json and the check exits 0; only a non-empty $out/unallowlisted-gated.json fails the build. Accepted deltas in nix/checks/proto-audit-netlink-allowlist.json. See netlink/coverage-status.md.
upstream-pins That nix/upstream-pins.json still matches the pins this build actually uses. Its networked counterpart, nix run .#check-upstream-pins, reports whether upstream main has moved — that half cannot be a check, because the nix flake check sandbox has no network.

The aggregated linter, audit, and coverage status is collected into quality-report.md by nix run .#update-quality-report.

Running the tests

go test ./...                                  # all unit tests locally
nix build .#test-go-unit                       # sandboxed unit run
nix build .#test-go-race                       # race detector
go test -bench=. ./pkg/xtcpnl/...              # benchmarks
nix build .#test-proto-deserialize-golden      # golden fixture decode
nix run  .#microvm-x86_64-lifecycle            # real-kernel integration test

See CONTRIBUTING.md for the full target list and integration-testing.md for the microVM harness.

See also