Skip to content

chore(infra): add scripts/lab, a Linux lab for macOS development - #5

Merged
oddurs merged 2 commits into
mainfrom
chore/linux-lab
Oct 3, 2026
Merged

oddurs merged 2 commits into
mainfrom
chore/linux-lab

Conversation

@oddurs

@oddurs oddurs commented Oct 3, 2026 •

Copy link
Copy Markdown
Owner

Problem

Development happens on macOS (Apple silicon), and everything liam ships is Linux: static musl binaries, io_uring, a kernel. scripts/task build:static refuses to run on macOS, and anything touching io_uring or the kernel had to wait for a CI round trip. Parallel builders also need a place to run those checks without sharing a target directory.

Approach

scripts/lab drives an OrbStack machine, liam-lab (Ubuntu 26.04, aarch64). OrbStack mounts /Users at the same path, so a worktree is the same directory on both sides and nothing is copied.

  • scripts/lab <cmd...> runs the command in the VM at the caller's directory, with PATH including ~/.cargo/bin and CARGO_TARGET_DIR=$HOME/.cache/liam-lab/target<worktree path>. Lab builds never touch the macOS target/, and each worktree gets its own directory.
  • scripts/lab setup is idempotent. It installs the packages the kernel and image work will need, rustup with the channel from rust-toolchain.toml and both musl targets, and writes ~/.cargo/config.toml in the VM.
  • Cross-compiling for x86_64 musl from aarch64 needs two things. The linker is x86_64-linux-gnu-gcc; Rust supplies musl's libc and start files. mimalloc's C needs x86_64 musl headers, and Ubuntu packages musl for the native architecture only. Plain x86_64-linux-gnu-gcc fails (bits/libc-header-start.h: No such file or directory). So setup builds musl 1.2.5 from upstream source for x86_64, with its musl-gcc wrapper, into ~/.local/x86_64-linux-musl. That is the version in Rust's bundled libc.a. The tarball is pinned by sha256, and the hash matches Ubuntu's musl_1.2.5.orig.tar.gz.
  • scripts/task build:static now reads ${CARGO_TARGET_DIR:-target} when it checks the binaries. That is the only change to a shared file, and CI behaves the same because it does not set the variable.
  • CONTRIBUTING.md has three lines for macOS contributors.

Evidence

All of it is in the notes on cairn item 0097.

  • scripts/lab cargo test --workspace: exit 0, all six test binaries ok.
  • scripts/lab scripts/task build:static: liamd 543,392 bytes and liam-init 397,880 bytes, both ELF 64-bit LSB pie executable, x86-64 ... static-pie linked, stripped. CI's static job gives 543,400 and 397,880 for the same code. Both binaries run in the lab under OrbStack's Rosetta binfmt, which exercises the cross-built mimalloc.
  • io_uring: a raw-syscall C program set up a ring, submitted IORING_OP_READ and reaped res=9 user_data=42. io_uring_disabled is 0 on kernel 7.0.14-orbstack.
  • Fresh provisioning: a throwaway ubuntu:resolute machine was set up from nothing by this script, with only the machine name changed, in 1m33s. build:static then passed on it. A second setup took 0.74s and reinstalled nothing. The throwaway machine was deleted.
  • Error paths: without orb on PATH, and with no liam-lab machine (a stub orb), the script exits 1 with the fix in the message. A command's exit status passes through.
  • scripts/task check is green on macOS.
  • A code review found two things that are now fixed in the second commit. The musl prefix is versioned, so a version bump rebuilds it. rustup-init is downloaded before it runs, so a failed fetch fails loudly. After the fix, a second throwaway machine was provisioned from nothing and build:static passed on it.
  • Item 0097 is closed in this branch. All seven criteria are ticked, with evidence in its notes.

What to look at sceptically

  • Mixed toolchains. The C code is compiled against musl 1.2.5 headers I built, and linked against Rust's bundled musl 1.2.5 libc.a. The versions match today. If Rust bumps its bundled musl, musl_version in scripts/lab should follow. CI's static job (native x86_64 musl-gcc) stays the authority for what ships.
  • The lab binaries are not byte-identical to CI's. liamd differs by 8 bytes because the gcc is different. Image size budgets should be measured in CI, not in the lab.
  • scripts/lab scripts/task check is not supported. site/node_modules lives in the shared worktree and belongs to macOS, and the VM has no pnpm. --help says to run check on macOS.
  • The review also noted that build:static reads only CARGO_TARGET_DIR, not CARGO_BUILD_TARGET_DIR or build.target-dir. I left it: nothing here sets those, and reading cargo's resolved target directory would need cargo metadata plus a JSON parser.
  • setup overwrites ~/.cargo/config.toml in the VM. The VM exists for this, but anything hand-added there will be lost.
  • Two setup runs at the same time would contend on apt's lock and on the musl prefix. Setup is a once-per-machine step, so I did not add locking.
  • The VM has no KVM, so Firecracker still runs only in CI.

Refs: 0097

oddurs added 2 commits October 2, 2026 21:16
Static musl builds, io_uring and kernel work are Linux-only, so on macOS
every such check waited on a CI round trip. scripts/lab runs a command in
the liam-lab OrbStack machine at the caller's directory, with a target
directory per worktree under the VM user's home, and scripts/lab setup
provisions the machine idempotently.

The machine is aarch64, so the x86_64 musl target links with the x86_64
cross gcc, and mimalloc's C compiles against an x86_64 musl 1.2.5 built
from upstream source (Ubuntu packages musl for the native architecture
only). build:static now honours CARGO_TARGET_DIR.

Refs: 0097
Review findings on the lab script. The musl prefix carried no version, so
bumping musl_version kept compiling against the old headers. rustup-init
was piped from curl, so a failed download ran an empty script, exited 0,
and setup then failed with a bare 'rustup: not found'.

Refs: 0097
@oddurs
oddurs merged commit 9b58ae0 into main Oct 3, 2026
3 checks passed
@oddurs
oddurs deleted the chore/linux-lab branch October 3, 2026 01:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant