Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions skills/tech-stack/references/practices/ci-runner-hosts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Runner hosts: tuning long-lived self-hosted runners

Extends the Runners section of [CI](ci.md). Measure first (see [CI options](ci-options.md)); a small host is harmed by advice written for large fleets.

- Before pruning build cache, check when each entry was last used (`docker buildx du --verbose`). "Reclaimable" in `docker system df` counts every entry no running container holds, including cache used minutes ago; pruning hot cache only slows the next builds. Prune with `--filter until=<age>`, and treat a prune that frees nothing as a sign that the cache is in use, not as a reason to drop the filter.
- Run every runner on a host as one service user. A runner per user copies the setup actions' tool cache, mise and rustup toolchains, package stores and a rootless Docker per user, and that duplication can outweigh everything the host's cleanup can free.
- On a long-lived self-hosted runner, install toolchains once on the host through mise, pinned by the project's `mise.toml`, rather than using `setup-*` actions in every job. Keep one pnpm store on the same filesystem as the runners' work folders so installs hardlink, and skip `actions/cache` there: on one disk it only adds an upload and a download.
- Route by job length with labels: one light runner for jobs under about a minute (change detection, aggregate gates, packaging) and the others for heavy jobs. On a small runner fleet, quick checks otherwise queue behind long test jobs. Fold tiny jobs into neighbours where the gate rules allow it, since each job pays runner start-up, checkout and install.
- Use `fetch-depth: 1` except in jobs that need history (versioning, releases, base-ref diffs), and sparse checkout for jobs that read a few paths. A job that needs history but only names (`merge-base`, `diff --name-only`, `log`) can add `filter: blob:none`. Never add it where the job reads old file contents (`git show`, content diffs, history secret scans) under `persist-credentials: false`: in a private repo, git fetches those blobs on demand with no credentials and the job fails.
- Do not run more concurrent jobs than the host's memory allows: on a 2-core, 4 GB host that is already swapping, more runners slow every job.
2 changes: 1 addition & 1 deletion skills/tech-stack/references/practices/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ verified: 2026-10-06 · https://docs.zizmor.sh
- Trigger on `labeled` and `unlabeled` as well, under the per-PR `concurrency` group, so switching reruns on the chosen runner and cancels the other run; job names stay the same, so required checks pass from either. Events from the workflow's `GITHUB_TOKEN` start no run or wait for manual approval, so a person or an app token applies the label. Both paths install tools through mise, so either runner gives the same result.
- Pick self-hosted runners by label, never by runner name: names can repeat across scopes. An organization's runners serve only its repositories; a personal repository needs its own runner or a move into the organization. A self-hosted runner gets a dedicated machine that runs nothing else; the project records its minimum size and why. When hosted minutes are blocked, follow the CI-unavailable rule in `hooks.md` rather than using a runner too small for the jobs.
- Keep the host's cleanup in version control (the project's infra repo) as one entry point on systemd timers: log what each step freed and fail visibly when disk stays low. Also run it on disk pressure (a check every few minutes, with a cooldown), not only on a schedule.
- A host that also runs deployments breaks the dedicated-machine rule; record it, and never run a global `docker image prune --all` there, since it [removes every image no container uses](https://docs.docker.com/reference/cli/docker/image/prune/), idle worker images included. Leave app-image retention to the deploy tool (Kamal's [`retain_containers`](https://kamal-deploy.org/docs/configuration/overview/), which [prunes after each deploy](https://kamal-deploy.org/docs/commands/deploy/)). `docker builder prune` has [no `--builder` flag](https://docs.docker.com/reference/cli/docker/builder/prune/) and a [`docker-container` builder](https://docs.docker.com/build/builders/drivers/docker-container/) (Kamal's default) keeps its cache in its own volume, so prune those with [`docker buildx prune --builder <name>`](https://docs.docker.com/reference/cli/docker/buildx/prune/).
- A host that also runs deployments breaks the dedicated-machine rule; record it, and never run a global `docker image prune --all` there, since it [removes every image no container uses](https://docs.docker.com/reference/cli/docker/image/prune/), idle worker images included. Leave app-image retention to the deploy tool (Kamal's [`retain_containers`](https://kamal-deploy.org/docs/configuration/overview/), which [prunes after each deploy](https://kamal-deploy.org/docs/commands/deploy/)). `docker builder prune` has [no `--builder` flag](https://docs.docker.com/reference/cli/docker/builder/prune/) and a [`docker-container` builder](https://docs.docker.com/build/builders/drivers/docker-container/) (Kamal's default) keeps its cache in its own volume, so prune those with [`docker buildx prune --builder <name>`](https://docs.docker.com/reference/cli/docker/buildx/prune/). Tuning a long-lived host (cache age, one runner user, preinstalled tools, light and heavy pools): [runner hosts](ci-runner-hosts.md).
- When `/tmp` is a small tmpfs, a [job-started hook](https://docs.github.com/en/actions/how-tos/manage-runners/self-hosted-runners/run-scripts) (`ACTIONS_RUNNER_HOOK_JOB_STARTED`) points `TMPDIR`, `TMP` and `TEMP` at `RUNNER_TEMP` through `$GITHUB_ENV`; the runner [empties `RUNNER_TEMP`](https://docs.github.com/en/actions/reference/workflows-and-actions/variables) itself. A hook's non-zero exit fails the job, so make the hook fail open: log and exit 0. Changing the hook path needs a runner restart, which kills the running job when the unit uses `KillMode=control-group` ([systemd's default](https://www.freedesktop.org/software/systemd/man/latest/systemd.kill.html); the runner's [stock unit](https://github.com/actions/runner/blob/main/src/Misc/layoutbin/actions.runner.service.template) uses `process`), so restart only when no job can be running.

## Base-ref gates
Expand Down
Loading