diff --git a/skills/tech-stack/references/practices/ci-runner-hosts.md b/skills/tech-stack/references/practices/ci-runner-hosts.md new file mode 100644 index 0000000..ba53a80 --- /dev/null +++ b/skills/tech-stack/references/practices/ci-runner-hosts.md @@ -0,0 +1,10 @@ +# Runner hosts: tuning long-lived self-hosted runners + +Extends the Runners section of [CI](ci.md). Measure first (see [CI options](ci-options.md)); a small host is harmed by advice written for large fleets. + +- Before pruning build cache, check when each entry was last used (`docker buildx du --verbose`). "Reclaimable" in `docker system df` counts every entry no running container holds, including cache used minutes ago; pruning hot cache only slows the next builds. Prune with `--filter until=`, and treat a prune that frees nothing as a sign that the cache is in use, not as a reason to drop the filter. +- Run every runner on a host as one service user. A runner per user copies the setup actions' tool cache, mise and rustup toolchains, package stores and a rootless Docker per user, and that duplication can outweigh everything the host's cleanup can free. +- On a long-lived self-hosted runner, install toolchains once on the host through mise, pinned by the project's `mise.toml`, rather than using `setup-*` actions in every job. Keep one pnpm store on the same filesystem as the runners' work folders so installs hardlink, and skip `actions/cache` there: on one disk it only adds an upload and a download. +- Route by job length with labels: one light runner for jobs under about a minute (change detection, aggregate gates, packaging) and the others for heavy jobs. On a small runner fleet, quick checks otherwise queue behind long test jobs. Fold tiny jobs into neighbours where the gate rules allow it, since each job pays runner start-up, checkout and install. +- Use `fetch-depth: 1` except in jobs that need history (versioning, releases, base-ref diffs), and sparse checkout for jobs that read a few paths. A job that needs history but only names (`merge-base`, `diff --name-only`, `log`) can add `filter: blob:none`. Never add it where the job reads old file contents (`git show`, content diffs, history secret scans) under `persist-credentials: false`: in a private repo, git fetches those blobs on demand with no credentials and the job fails. +- Do not run more concurrent jobs than the host's memory allows: on a 2-core, 4 GB host that is already swapping, more runners slow every job. diff --git a/skills/tech-stack/references/practices/ci.md b/skills/tech-stack/references/practices/ci.md index 8599c88..1a2f286 100644 --- a/skills/tech-stack/references/practices/ci.md +++ b/skills/tech-stack/references/practices/ci.md @@ -25,7 +25,7 @@ verified: 2026-10-06 ยท https://docs.zizmor.sh - Trigger on `labeled` and `unlabeled` as well, under the per-PR `concurrency` group, so switching reruns on the chosen runner and cancels the other run; job names stay the same, so required checks pass from either. Events from the workflow's `GITHUB_TOKEN` start no run or wait for manual approval, so a person or an app token applies the label. Both paths install tools through mise, so either runner gives the same result. - Pick self-hosted runners by label, never by runner name: names can repeat across scopes. An organization's runners serve only its repositories; a personal repository needs its own runner or a move into the organization. A self-hosted runner gets a dedicated machine that runs nothing else; the project records its minimum size and why. When hosted minutes are blocked, follow the CI-unavailable rule in `hooks.md` rather than using a runner too small for the jobs. - Keep the host's cleanup in version control (the project's infra repo) as one entry point on systemd timers: log what each step freed and fail visibly when disk stays low. Also run it on disk pressure (a check every few minutes, with a cooldown), not only on a schedule. -- A host that also runs deployments breaks the dedicated-machine rule; record it, and never run a global `docker image prune --all` there, since it [removes every image no container uses](https://docs.docker.com/reference/cli/docker/image/prune/), idle worker images included. Leave app-image retention to the deploy tool (Kamal's [`retain_containers`](https://kamal-deploy.org/docs/configuration/overview/), which [prunes after each deploy](https://kamal-deploy.org/docs/commands/deploy/)). `docker builder prune` has [no `--builder` flag](https://docs.docker.com/reference/cli/docker/builder/prune/) and a [`docker-container` builder](https://docs.docker.com/build/builders/drivers/docker-container/) (Kamal's default) keeps its cache in its own volume, so prune those with [`docker buildx prune --builder `](https://docs.docker.com/reference/cli/docker/buildx/prune/). +- A host that also runs deployments breaks the dedicated-machine rule; record it, and never run a global `docker image prune --all` there, since it [removes every image no container uses](https://docs.docker.com/reference/cli/docker/image/prune/), idle worker images included. Leave app-image retention to the deploy tool (Kamal's [`retain_containers`](https://kamal-deploy.org/docs/configuration/overview/), which [prunes after each deploy](https://kamal-deploy.org/docs/commands/deploy/)). `docker builder prune` has [no `--builder` flag](https://docs.docker.com/reference/cli/docker/builder/prune/) and a [`docker-container` builder](https://docs.docker.com/build/builders/drivers/docker-container/) (Kamal's default) keeps its cache in its own volume, so prune those with [`docker buildx prune --builder `](https://docs.docker.com/reference/cli/docker/buildx/prune/). Tuning a long-lived host (cache age, one runner user, preinstalled tools, light and heavy pools): [runner hosts](ci-runner-hosts.md). - When `/tmp` is a small tmpfs, a [job-started hook](https://docs.github.com/en/actions/how-tos/manage-runners/self-hosted-runners/run-scripts) (`ACTIONS_RUNNER_HOOK_JOB_STARTED`) points `TMPDIR`, `TMP` and `TEMP` at `RUNNER_TEMP` through `$GITHUB_ENV`; the runner [empties `RUNNER_TEMP`](https://docs.github.com/en/actions/reference/workflows-and-actions/variables) itself. A hook's non-zero exit fails the job, so make the hook fail open: log and exit 0. Changing the hook path needs a runner restart, which kills the running job when the unit uses `KillMode=control-group` ([systemd's default](https://www.freedesktop.org/software/systemd/man/latest/systemd.kill.html); the runner's [stock unit](https://github.com/actions/runner/blob/main/src/Misc/layoutbin/actions.runner.service.template) uses `process`), so restart only when no job can be running. ## Base-ref gates