feat(root-ubuntu): add zram + earlyoom so a memory spike stalls, not wedges

[ralyodio]

Why

configure_swap gives a box slack. Nothing decided what dies when the slack runs out, and on its own it bought that slack in the most expensive currency the box has: disk.

The failure this prevents reads as a hardware fault. RAM fills → the kernel reclaims → reclaim goes to disk → every process blocks on that disk → load climbs → the kernel's OOM killer (which only fires once the machine is already thrashing) shoots something large and arbitrary. From a terminal it is indistinguishable from a reboot — and uptime afterwards says it wasn't one, which tends to get believed over the person who watched it happen.

This is not hypothetical: it hit dev twice in one night, and two of our public boxes were running with zero swap.

What

Two functions, fixing different halves:

zram	compressed block device in RAM, used as swap ahead of the file	reclaim costs CPU instead of seeks. ~3:1 with zstd — a live box here holds 468 M of pages in 170 M. It doesn't add memory, it postpones the wall.
earlyoom	kills the biggest consumer while the box is still responsive	the kernel's killer isn't wrong about what to kill, it's late

Ordered after configure_swap, which returns early if any swap is active — zram first would suppress the swapfile entirely. The file stays as the deeper tier; zram sits above it at priority 100.

Idempotent

Config goes through write_if_changed; the service is bounced only when something changed or isn't running; and an already-correct zram device is never cycled — doing that on every --refresh would drop whatever the box had paged out.

Verified by running both functions twice on a live box:

RUN 1   earlyoom watching (TERM at 10% free, KILL at 5%)
        zram swap (min(ram / 2, 8192) cap, zstd) ahead of /swapfile
        vm.swappiness=100
        CHANGED=3

RUN 2   earlyoom already watching
        zram already active:  7.8G zstd
        CHANGED=0

Three traps handled, all of which fail silently or half-way

1. DigitalOcean images ship no zram module. modprobe zram fails, and the unit then blames dev-zram0.device — never naming the real cause. Installs linux-modules-extra-$(uname -r) and retries. (All four of our DO droplets needed this; netcup did not.)
2. A zram0 that already has a disksize refuses reconfiguration with Device or resource busy and quietly keeps its old size and algorithm — so a changed config appears to apply and hasn't. Reset before restart.
3. systemd expands $EARLYOOM_ARGS from EnvironmentFile by splitting on whitespace with no shell quote removal, so a quoted --avoid '^(sshd|...)$' arrives with the quote characters inside the pattern and matches nothing. The service starts happily either way. Documented in-line, verify with ps -o args= -C earlyoom.

Notes

• vm.swappiness=10 is right for a disk swapfile and wrong once paging out is a memcpy, so a 61- drop-in raises it to 100 — but only where zram actually came up. A box with just a swapfile keeps the conservative value.
• EARLYOOM_PREFER is deliberately empty by default. A prefer list is right on a workstation (the answer is obviously "the browser"); on a server the biggest consumer is the app, and shooting it first only means it dies sooner than the box would have made it. Let size decide and let systemd restart it.
• Skipped inside containers, same reasoning as the existing swapfile.
• bash -n clean; no new shellcheck warnings in the added section.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SRvWqCAeKAvvHFmQom8iMT

[github-actions]

ThreatCrush Security Scan

10 finding(s)

HIGH/CRITICAL: 2 | MEDIUM: 4 | LOW: 4

Severity	Rule	Location
HIGH	sh-remote-script-execution	root-ubuntu.sh:3004
HIGH	sh-remote-script-execution	root-ubuntu.sh:3008
MEDIUM	sh-remote-script-execution	root-ubuntu.sh:3157
MEDIUM	sh-remote-script-execution	root-ubuntu.sh:4186
MEDIUM	redos-nested-quantifier	src/domain-free.ts:56
MEDIUM	redos-nested-quantifier	src/mail.ts:1042
LOW	secret-generic-credential	src/credentials.ts:36
LOW	secret-generic-api-key	test/credentials.test.ts:208
LOW	secret-generic-credential	test/mail.test.ts:135
LOW	secret-generic-credential	test/shorten.test.ts:36

Snippets are redacted; ThreatCrush never prints matched credential material.