Bake the nixpkgs git cache into the VM image: first harness install 36s → 11s, zero per-user disk

[lionello]

Summary

A box's first JIT harness install spends ~25 of its ~36 seconds doing nothing that is specific to the harness: it ingests the pinned nixpkgs tarball into nix's per-user git cache. That work is identical on every box, for every harness, and for every user — and it is ~72 MiB that a VM image could carry.

Baking tarball-cache-v2 into the image and hardlinking it into each user's ~/.cache/nix at supervisor start takes first-use harness install from ~36s to ~11s, costs 0 bytes of per-user disk, and does not bake any harness closure — so JIT stays JIT and the disk-space reason for JIT is preserved.

Complements #636. That issue stops a first-use fetch from blocking the reconcile loop; this one makes the fetch 3× shorter. Neither subsumes the other.

Measurements

Clean Azure Standard_* aarch64 box (2 vCPU), Determinate Nix 3.22.3 / 2.35.2, claude-code-2.1.266, store diffed before and after the install.

Phase budget for the observed install:

phase	time
download nixexprs.tar.xz (40 MiB)	0.45s
xz-decompress + git-ingest into ~/.cache/nix/tarball-cache-v2	~25s
evaluate nixpkgs → outPath	1.1s
substitute 33 build-time deps (26 MiB)	~2s
download claude.zst (72 MiB)	~0.8s
unpack + autoPatchelf + wrap → 206 MiB output	~4s
write profile generation	~2s

The 25s is not decompression — that is 1.6s on its own:

$ time curl -sS "$AGENT_BOX_NIXPKGS" | xz -dc | wc -c
279418880
real	0m1.623s

It is git object construction: 54,075 blobs SHA-1'd and deflated, 38,177 trees, 92,253 objects into 3 packs, CPU-bound (user 36.7s vs real 26.7s — both cores pinned). Phase boundary, from timestamped nix eval -v on a cold XDG_CACHE_HOME:

[  0.041s] unpacking 'nixexprs.tar.xz' into the Git cache...
[ 25.493s] evaluating file '«…»/flake.nix'
[ 26.994s] /nix/store/2386529…-claude-code-2.1.266

Same eval with the cache warm: 1.1s.

Why this is safe to bake

agent_install installs from an immutable, pinned URL that is already exported into every session, so the image build knows exactly which revision to bake — guaranteed hit, no heuristic:

AGENT_BOX_NIXPKGS=https://releases.nixos.org/nixos/unstable/nixos-26.11pre1070770.8ce4ef6cb6f8/nixexprs.tar.xz
  → nix profile add --impure "$AGENT_BOX_NIXPKGS#$_ai_attr"

Source: agent_install.

It is also the same revision for every harness, so one 72 MiB artifact accelerates claude, codex and anything added to agent_names later. The cost is per-user-first-install, not per-harness.

What to bake — 72 MiB, and both parts are required

~/.cache/nix/tarball-cache-v2/objects/pack/*    69 MiB   nixpkgs git objects
~/.cache/nix/fetcher-cache-v4.sqlite           124 KiB   URL → treeHash → narHash map

Baking the packs without the sqlite does not work: nix cannot learn the tree hash without redoing the ingest, and re-ingests in 23.7s. The 124 KiB is load-bearing.

Distribution: hardlink, not alternates

The cache is per-user. Two mechanisms measured:

mechanism	eval	new disk per user
hardlink the packfiles into ~/.cache/nix/	1.28s ✅	0 bytes (verified: link count 2)
objects/info/alternates → shared read-only object store	20.2s ❌	—

Negative result worth recording so nobody else spends an afternoon on it: git alternates do not work here. Nix's libgit2 path ignored objects/info/alternates and re-ingested from scratch.

Hardlinking is free and correct: packfiles are immutable and mode 0444, so every user's cache points at the same inodes and consumes no additional blocks. The sqlite is mutable and must be a real copy.

Proposed implementation

No change to the apply/user-creation path, and no new backend-specific code — the supervisor already scaffolds per-user $HOME on every start, with exactly the right idempotent, best-effort idiom:

# Ship ~/worktrees, empty (issue #126).
mkdir -p "$HOME"/worktrees 2>/dev/null || :

modules/src/supervisor.sh#L92-L103

Add, alongside it, a best-effort seed of the nixpkgs cache from a baked location (e.g. /var/lib/agent-box/nixpkgs-cache/), when that location exists and the user's cache does not already hold it:

• cp -al <baked>/tarball-cache-v2 "$HOME"/.cache/nix/ (hardlink; falls back to cp -a across filesystems)
• cp -a <baked>/fetcher-cache-v4.sqlite "$HOME"/.cache/nix/ (real copy)

Same failure posture as the mkdir above: a box with no baked cache is exactly today's behaviour, so this is inert on every existing deployment and needs no migration.

The image build then only has to run one nix eval against $AGENT_BOX_NIXPKGS and copy the resulting cache to the baked location.

It degrades gracefully when the pin moves

The usual objection to baking is staleness. Measured, using nixos-unstable-small as a stand-in for a moved pin:

empty cache, different rev          23.8s
baked pack present, different rev    7.3s     ← 3.3× faster
cache holding both revs              78 MiB   ← vs 72 MiB for one

Git dedupes ~92% of objects, so an image that has fallen behind the pin keeps most of the benefit, and holding two revisions costs 6 MiB. Baking the pack is never wrong, only more or less useful — a much better risk profile than baking harness closures, which are all-or-nothing per version and which this issue explicitly does not propose.

Root pays the same tax at provisioning

First boot runs nix profile install github:defangdevs/agent-box/<rev>#runtime, which fetches nixpkgs as a flake input and ingests it again under /root. Bootstrap on this box was 4m11s wall (boot 15:40:50 → bootstrap complete 15:45:01, /var/log/agent-box-bootstrap.log). Seeding root's cache from the same baked artifact removes ~25s of that too.

Notes

• tarball-ttl = 3600, so an image older than an hour makes one conditional GET on first use. The URL is an immutable versioned release, so it is a 304 in ~0.25s. Nothing to fix; noted so it does not confuse anyone reading a trace.
• The classic non-flake path is not an escape hatch: nix-instantiate "$AGENT_BOX_NIXPKGS" -A claude-code routes through the same fetcher, builds the same pack, and copies nixpkgs into the store on top — 51s, strictly worse.
• claude-code is meta.license.free = false, so Hydra never builds it; cache.nixos.org and defanglabs.cachix.org both 404 on its narinfo. Substituting the built closure would be faster still (measured: 2.0s to restore the full 306 MiB closure into an empty store) but requires a CI job pushing per (harness version × nixpkgs rev). This issue deliberately proposes the cheaper, more robust half.

Acceptance

• A box provisioned from an image carrying the baked cache completes its first harness install in materially less time than one without it, with no harness closure present in the image.
• A user whose ~/.cache/nix is seeded by hardlink consumes no additional disk for the packfiles (link count > 1).
• A box with no baked cache present behaves exactly as today; no migration, no new failure mode when the location is absent or unreadable.
• A baked cache whose revision no longer matches $AGENT_BOX_NIXPKGS still completes, correctly, without manual intervention.

P2: no availability impact; this is provisioning and first-use latency, and the current behaviour is correct just slow.

[defangdevs]

Triaged this rather than shipping straight to a PR, because the literal proposal ("bake into the VM image") forks into a real infra decision that's worth deciding explicitly rather than picking silently.

There is no existing custom VM-image build in this repo, for either cloud. AWS tracks stock upstream NixOS AMIs (scripts/refresh_amis.py just rewrites deploy/aws/template.yaml's region map from nixos.github.io/amis/images.json weekly), and Azure deploys from deploy/azure/agent-box.bicep against a marketplace image. Both provision entirely at first boot via nix profile install github:defangdevs/agent-box/<rev>#runtime (nix/runtime.nix). So "the image build" this issue's implementation section refers to doesn't exist yet — adding it means a new build+publish pipeline per cloud (AWS: build, copy to 4 regions, wire into the AMI-map refresh; Azure: build, publish a managed image), which is a materially bigger and more security-sensitive change than the supervisor.sh hunk the issue sketches.

Second gap: the pin isn't always immutable. AGENT_BOX_NIXPKGS (jitNixpkgsRef in modules/agent-box.nix:2833-2836) is only pinned to an immutable release URL when selfUpdate.agentNixpkgs is wired. The documented fallback (same file, ~line 11750) is a mutable nixos-unstable channel URL, deliberately — "Pinning it by DEFAULT would need its own hash-maintenance path and is deliberately out of scope." A baked cache keyed to one rev only pays off on the subset of the fleet that has that pin configured; worth being explicit that this is an optimization for the pinned fleet, not universal.

A smaller alternative gets most of the win with none of the new image infra: root already runs nix profile install .../#runtime at first boot (nix/runtime.nix), which ingests some nixpkgs revision into root's own ~/.cache/nix right then. Instead of baking into an image:

• After that first-boot install, seed a shared /var/lib/agent-box/nixpkgs-cache/ (from that ingest, or a one-time nix eval against the box's live jitNixpkgsRef).
• In supervisor.sh, right alongside the existing mkdir -p "$HOME"/worktrees best-effort seeding (modules/src/supervisor.sh ~L92-103, the same spot this issue's own proposal points at), hardlink-seed each new user's ~/.cache/nix from that shared location the same way this issue already proposes.

That satisfies every acceptance criterion here except accelerating the very first root ingest on box boot — which happens once per box, not once per user, so it's a much smaller miss than it sounds. No new image-build/publish pipeline, no multi-region/multi-cloud replication, no separate staleness story (the box regenerates the shared cache itself from whatever it's actually pinned to, so it's never stale relative to itself).

Given the cost difference, does the simpler "seed from the box's own first-boot cache" direction cover what you're after, or is baking into the image itself the point (e.g. to also cut first-boot-to-ready wall time before any user session exists)? Happy to build either once that's settled — the supervisor.sh half is identical in both.

[defangdevs]

Done, in two halves, and the split matters.

The on-box producer landed in #695. agent-box-nixpkgs-cache.service (driven by a timer) builds /var/lib/agent-box/nixpkgs-cache from the ref the box is actually pinned to, and supervisor.sh's seed from #670 — inert on every box since it merged, because nothing had ever written that directory — finally has something to read.

Measured end to end on an aarch64 box, real nix, real pinned ref: a new user's first claude-code resolve goes from 27.6s to 1.4s, and the producer is a 15ms no-op on every boot after the first.

Two things this issue proposed turned out to be wrong, both recorded in the code:

• The hardlink cannot save disk on a real box. Sessions run under ProtectSystem=strict with ReadWritePaths=$HOME, which makes the home its own bind mount, and link(2) refuses to cross a mount point even on one filesystem — so cp -al from /var/lib always falls through to the copy arm. The "zero bytes per user, link count 2" measurement in perf(sessions): seed nix's nixpkgs git cache from the image #670 was taken outside the unit's namespace. Real cost is ~72 MiB per user; only root, which is outside that namespace, could make the original claim true.
• The guard has to key on the pin, written last. Publishing it earlier left, on any interruption during a pin move, the old cache wearing the new pin — which the guard reads as "already current", so the new cache would never be published and no later run would revisit it.

The image-baking half is #697, where it belongs: there is no custom VM-image build in this repo for either cloud, so "bake it into the image" means inventing a build-and-publish pipeline per cloud. That work is under way for Azure (#698 landed the pipeline), and it is a much bigger lever than this issue measured — the bootstrap is 3m47s, of which a single 797 MiB nix profile install is ~750 of its 855 log lines, and that same install takes 2.9s against a warm store.

Closing this one: the producer exists, and the remaining ask is tracked in #697.