feat(update): agentbox rollback, the undo for an update that succeeded (#676)

[lionello]

First of the four pieces in #676. Standalone and useful on its own: it closes the gap that has no workaround today.

What is missing

Every rollback path in the updater lives inside the update transaction - a build that fails, a profile switch that fails, an apply that raises, a release that cannot run its own post-switch phase (cmd_update, post_switch, recover_handover). None of them can see the failure this verb is for: a release that applies cleanly, restarts every unit, returns 0, and is wrong anyway. restart_units() never asks whether the daemons came back, and the code says outright that "apply is not a health check with a timeout" - so the moment post_switch returns 0 the release is permanent. Until this, the only way back was a root shell.

The verb

sudo -n /usr/bin/systemctl start --no-block agent-box-rollback.service

Switches the runtime profile back to the previous release, resets the source tree to the rev that generation names, re-applies with THAT release's own agentbox, and restarts the services. agentbox rollback --check names the target and touches nothing.

Going back is a profile generation, not a git revert. Generation N-1 is already realized in the store and already a GC root, so the undo is rollback_to() plus an apply. Reverting the tree instead would move the record without moving the closure, which since #242 is backwards - the tree says what the profile was built from, and the profile is what the box runs. It also avoids spending --force: returning to the rev behind the current one is a downgrade in git's terms and the fast-forward guard would refuse it, so an update-based undo would have to skip the one check that stops a replay of an older rev.

The bug the live box caught

The target is the newest generation below the current one that actually holds a release, not simply N-1. cmd_update installs by removing the runtime element and adding it back, and nix profile remove makes a generation of its own - so every successful update leaves an empty generation directly behind the release it installed.

Measured on a live native box while writing this (agent@20-237-183-155.sslip.io, running 770ef4e):

generation 1  ->  github:defangdevs/agent-box/1c97fd8  (the install)
generation 2  ->  "elements": []                       (nix profile remove)
generation 3  ->  git+file:///var/lib/agent-box/src?rev=770ef4e  (running)

The first draft of this trusted N-1 and refused with "generation 2 has no bin/agentbox" against that profile. Had the check not been there, it would have switched the box onto a profile with no runtime and no agentbox to put it back. holds_a_release() and previous_release() are that fix, and the walk skips a half-collected generation for the same reason.

A real limit worth stating: agent-box-nix-gc.timer runs nix-collect-garbage --delete-older-than 7d, which deletes old generations and their closures. A box that has not updated inside that window has nothing to go back to. The command refuses rather than half-doing it, and says why. Piece 3 of #676 (the commit window) should hold its own GC root for the candidate; that is out of scope here.

Scope and one-sidedness

Native only, like the mechanism - a NixOS box has no runtime profile to walk, so its equivalent is a system generation (nixos-rebuild switch --rollback) and a different unit. The capability gap is real and one-sided though, so it is declared in UNITS_KNOWN_GAPS and SUDOERS_KNOWN_GAPS rather than the by-design tables, owned by #676.

The sudo grant is implied by the same rule that implies the update trigger, deliberately: an agent that can move this box forward and cannot move it back has the half of the pair that breaks boxes. Both units are root oneshots with a fixed ExecStart.

Coverage

15 cases in tests/test_agentbox.py:

• happy path - the restored release's own agentbox re-applies (named through the profile symlink, which is why the rollback has to happen first: the same path resolves to a different release either side of it), the source tree comes back with the profile, --check changes nothing, --to-generation names an older release exactly.
• refusals - not a generation symlink, nothing behind generation 1, a forward --to-generation, a collected generation, a --to-generation holding no release.
• partial failures - a tree that cannot be reset is a note and not a failed rollback (the box is already back on the release that works), a failed apply after the switch is reported, a rollback that will not move re-applies nothing.
• the empty-generation skip has its own regression test carrying the live measurement above.

Checks run

On aarch64, all passing: python3 tests/test_agentbox.py (165 tests, OK), scripts/check_backend_parity.py, scripts/check_one_spec.py, scripts/check_vendor.py, tests/test-assemble-module.py, bin/assemble-module.py --check (no module drift - nothing under modules/src changed, so tests/golden/ does not move either), tests/test-source-tree.sh, tests/test-checkout-bootstrap.sh. tests/native/expected regenerated. The VM lanes are x86_64-only and run in CI.

One thing found while running these, not fixed here: tests/test_agentbox.py fails on any box that has /etc/agent-box/agent-nixpkgs-pin - Spec reads that host path, so the rendered AGENT_BOX_NIXPKGS picks up the live box's resolved channel release instead of the fixture's channels.nixos.org URL. It passes in the Nix sandbox and fails for anyone running it on a real box, which is the population most likely to run it while debugging. Two tests, test_matches_committed_fixture and test_both_backends_expose_the_same_jit_pin_knob, both red on unmodified master here. Worked around by pointing AGENT_PIN_FILE at a nonexistent path while regenerating; worth its own change.

Fixes nothing on its own - #676 stays open for pieces 2 through 4.

[coderabbitai]

Warning

Review limit reached

Next included review available in 18 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: c9f4366f-f7e8-492a-9201-85c1a262a532

📥 Commits

Reviewing files that changed from the base of the PR and between 4ee19aa and c608f5d.

📒 Files selected for processing (9)

• bin/agentbox
• scripts/check_backend_parity.py
• scripts/check_one_spec.py
• tests/native/expected-modes.json
• tests/native/expected/etc/agent-box-guides/AGENTS.agent.md
• tests/native/expected/etc/agent-box-guides/AGENTS.robot.md
• tests/native/expected/etc/sudoers.d/agent-box
• tests/native/expected/etc/systemd/system/agent-box-rollback.service
• tests/test_agentbox.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}