From 57cc8e6b34814d1731ee94432b988114f85665e4 Mon Sep 17 00:00:00 2001 From: Derek Date: Mon, 3 Aug 2026 14:47:03 +1000 Subject: [PATCH] fix: separate the desktop user from the user running the install The playbook assumed the two were the same. On a fleet machine they are not: hyperi-infra's inventory connects as `ubuntu` while the desktop belongs to `hyperi`, and everything user-scoped landed in the wrong home or failed outright. `-e hyperi_target_user=` names the desktop user; `actual_user` and `user_home` derive from it. Molecule verify takes it as MOLECULE_TARGET_DESKTOP_USER, since `molecule verify` accepts no extra arguments, and now prints which user it checked -- a run against the wrong home was previously indistinguishable from a clean one. `become: false` means the CONNECTING user, so every user-scoped task carrying it was acting on the wrong account. An audit of tasks combining `become: false` with `user_home` found 19, of which 8 were wrong; the rest are macOS-only, where Homebrew refuses to run as root and the two users cannot differ. Two of the eight paired `become: false` with `become_user:`, which does nothing -- `become_user` is ignored while `become` is false. The worst of them made a tombstone lie. `ansible.builtin.file` cannot read a path it has no permission for and reports "Insufficient permissions ... Treating as absent", so the retired `~/.cargo/bin/uv` was reported removed while it sat there shadowing the real install on PATH. The molecule verify agreed, because it stat'ed the service account's empty home. Also removes the Docker Desktop context, which the package uninstall leaves behind: context state is a per-user file under ~/.docker/contexts, not package state. Where `desktop-linux` was also the selected context every docker command failed against a socket that no longer exists. Two hyperi-update fixes, both of which made it a no-op where it mattered most: - It gated on `sudo -v`, and Ubuntu 25.10+ ships sudo-rs, whose `-v` demands interactive authentication even where NOPASSWD grants the commands. It aborted before doing anything on every 26.04 box and under every unattended caller, which is what --yes exists for. Probes with `sudo -n true` now. - It did not wait for the dpkg lock, so an unattended-upgrades run in progress made the upgrade exit 100 while everything else reported success -- a box that looks updated and is not. Adds a watchdog to the rdp-server role for a fault that makes a remote-desktop host permanently unreachable. Force-stopping another session from the RDP login screen makes gdm tear down that session and the in-flight RDP login with it, and leaves the system daemon accepting TCP on 3389 while servicing nothing -- later connections get a blank screen and the daemon does not log so much as an incoming connection. Restarting gdm does not clear it; only restarting gnome-remote-desktop does, which on a headless host needs ssh that may not exist. The watchdog matches gdm's own failure line rather than probing the port: telling a wedged daemon from a healthy one over the wire needs a real RDP connection, and an aborted one exercises the same session teardown that wedges it, so a probe risks causing the fault it watches for. Gated on the presence of the GNOME handover unit, which is the mechanism the fault lives in, rather than on a version number. --- ansible/molecule/README.md | 14 ++++ ansible/molecule/existing-host/verify.yml | 62 +++++++++++++- ansible/playbooks/main.yml | 10 ++- ansible/roles/astral/tasks/main.yml | 11 ++- ansible/roles/developer-python/tasks/main.yml | 3 +- .../files/update/hyperi-update-linux.sh | 35 +++++++- ansible/roles/developer/tasks/nodejs.yml | 12 ++- ansible/roles/developer/tasks/removals.yml | 54 +++++++++++- ansible/roles/developer/tasks/verify.yml | 2 +- ansible/roles/rdp-server/README.md | 75 +++++++++++++++++ ansible/roles/rdp-server/defaults/main.yml | 9 ++ .../rdp-server/files/rdp-handover-watchdog | 58 +++++++++++++ .../files/rdp-handover-watchdog.service | 19 +++++ .../rdp-server/tasks/handover_watchdog.yml | 83 +++++++++++++++++++ ansible/roles/rdp-server/tasks/main.yml | 10 +++ ansible/roles/soe/tasks/claude.yml | 2 +- ansible/roles/soe/tasks/verify.yml | 6 +- docs/install-matrix.md | 2 +- 18 files changed, 450 insertions(+), 17 deletions(-) create mode 100644 ansible/roles/rdp-server/files/rdp-handover-watchdog create mode 100644 ansible/roles/rdp-server/files/rdp-handover-watchdog.service create mode 100644 ansible/roles/rdp-server/tasks/handover_watchdog.yml diff --git a/ansible/molecule/README.md b/ansible/molecule/README.md index 543cbc9..c30944e 100644 --- a/ansible/molecule/README.md +++ b/ansible/molecule/README.md @@ -96,6 +96,20 @@ it. molecule converge -s existing-host -- --tags soe molecule verify -s existing-host +**When the connection user is not the desktop user**, name the desktop one -- +a fleet machine is reached as a service account whose home holds none of the +artefacts under test: + + export MOLECULE_TARGET_USER=ubuntu # who we ssh as + export MOLECULE_TARGET_DESKTOP_USER=hyperi # whose machine it is + molecule converge -s existing-host -- --tags soe -e hyperi_target_user=hyperi + molecule verify -s existing-host + +Without it every user-scoped check passes against the service account's empty +home while the real user keeps the artefact -- a green run over a host that was +never fixed. `converge` takes it as `-e hyperi_target_user`; `verify` takes no +extra arguments, so it reads the environment. + `--tags soe` (or `--tags removals`) is not optional for a remediation run: the tombstones gate on `ansible_run_tags`, so a plain converge installs the new tools and removes nothing. diff --git a/ansible/molecule/existing-host/verify.yml b/ansible/molecule/existing-host/verify.yml index 8fbe60c..be79494 100644 --- a/ansible/molecule/existing-host/verify.yml +++ b/ansible/molecule/existing-host/verify.yml @@ -14,17 +14,42 @@ hosts: all gather_facts: true + vars: + # The DESKTOP user, which on a fleet machine is not the user ansible + # connected as. Checking the service account's empty home passes every + # user-scoped assert below while the real user still carries the artefact, + # so this is supplied the same way the inventory takes its host: + # MOLECULE_TARGET_DESKTOP_USER=hyperi molecule verify -s existing-host + verify_user: >- + {{ lookup('env', 'MOLECULE_TARGET_DESKTOP_USER') + | default(hyperi_target_user | default(ansible_user_id, true), true) }} + tasks: # ------------------------------------------------------------------ # SHADOWING — a stale user-level binary ahead of the system one on PATH # ------------------------------------------------------------------ + - name: Resolve the desktop user's home + ansible.builtin.getent: + database: passwd + key: "{{ verify_user }}" + become: true + + - name: Set the home to check + ansible.builtin.set_fact: + verify_home: "{{ ansible_facts.getent_passwd[verify_user][4] }}" + + - name: Report which user is being verified + ansible.builtin.debug: + msg: "Verifying user-scoped state for {{ verify_user }} ({{ verify_home }})" + - name: Stat the retired user-level uv locations ansible.builtin.stat: path: "{{ item }}" loop: - - "{{ ansible_env.HOME }}/.cargo/bin/uv" - - "{{ ansible_env.HOME }}/.cargo/bin/ruff" + - "{{ verify_home }}/.cargo/bin/uv" + - "{{ verify_home }}/.cargo/bin/ruff" register: verify_stale_uv + become: true - name: Assert no stale user-level uv shadows the system install ansible.builtin.assert: @@ -83,6 +108,37 @@ loop_control: label: "{{ item.item }}" + # A docker context is a per-user file, not package state, so it survives + # the uninstall. Left selected, it points at a dead socket and every docker + # command fails. + - name: Read the docker contexts + ansible.builtin.command: docker context ls --quiet + register: verify_docker_ctx + changed_when: false + failed_when: false + become: true + become_user: "{{ verify_user }}" + + - name: Assert the retired Docker Desktop context is gone + ansible.builtin.assert: + that: + - "'desktop-linux' not in (verify_docker_ctx.stdout_lines | default([]))" + fail_msg: >- + The desktop-linux context is still registered. It points at Docker + Desktop's socket, which no longer exists — and if it is also the + selected context, every docker command on this host fails. + success_msg: "no retired Docker Desktop context" + when: verify_docker_ctx.rc == 0 + + - name: Assert the selected docker context is not the retired one + ansible.builtin.command: docker context show + register: verify_docker_ctx_current + changed_when: false + failed_when: verify_docker_ctx_current.stdout | trim == 'desktop-linux' + become: true + become_user: "{{ verify_user }}" + when: verify_docker_ctx.rc == 0 + # ------------------------------------------------------------------ # RETIRED PACKAGES — dropped by decision, must not linger # ------------------------------------------------------------------ @@ -166,7 +222,7 @@ # ------------------------------------------------------------------ - name: Read ~/.zprofile if present ansible.builtin.slurp: - src: "{{ ansible_env.HOME }}/.zprofile" + src: "{{ verify_home }}/.zprofile" register: verify_zprofile failed_when: false diff --git a/ansible/playbooks/main.yml b/ansible/playbooks/main.yml index ab90800..0042625 100644 --- a/ansible/playbooks/main.yml +++ b/ansible/playbooks/main.yml @@ -139,9 +139,17 @@ (ansible_facts['distribution'] == 'Ubuntu' and ansible_facts['distribution_version'] is version('24.04', '<')) tags: ['always'] + # Whose machine this is. Normally the user running the install, but a fleet + # machine is reached as a service account (hyperi-infra connects as + # `ubuntu`) whose home is not the desktop, so `-e hyperi_target_user=` + # names the real one. Everything user-scoped keys off this: shell config, + # uv, Claude Code, dconf, and the tombstones. - name: Detect actual user (for sudo operations) ansible.builtin.set_fact: - actual_user: "{{ ansible_facts['env'].SUDO_USER | default(ansible_facts['user_id'], true) }}" + actual_user: >- + {{ hyperi_target_user + | default(ansible_facts['env'].SUDO_USER + | default(ansible_facts['user_id'], true), true) }} tags: ['always'] - name: Set actual user home directory (Linux) diff --git a/ansible/roles/astral/tasks/main.yml b/ansible/roles/astral/tasks/main.yml index 127ac58..817034b 100644 --- a/ansible/roles/astral/tasks/main.yml +++ b/ansible/roles/astral/tasks/main.yml @@ -73,6 +73,9 @@ dest: "/tmp/uv-{{ uv_arch }}.tar.gz" mode: '0644' + # Check mode only previews the directory and the download above, so on a + # host that has never had uv this fails the whole run instead of + # previewing it. - name: Extract uv ansible.builtin.unarchive: src: "/tmp/uv-{{ uv_arch }}.tar.gz" @@ -83,6 +86,7 @@ group: "{{ actual_user }}" extra_opts: - --strip-components=1 + when: not ansible_check_mode - name: Remove uv tarball ansible.builtin.file: @@ -115,12 +119,17 @@ # Ubuntu: no apt repo; install as user-owned uv tools (hyperi-update refreshes # via `uv tool upgrade`). uv is installed above, so it is on PATH here. +# +# Runs as the owner of that home, not as whoever connected: `become: false` +# means the connection user, and on a fleet machine that is a service account +# with no write access to the desktop user's home. - name: Install ruff and ty (Ubuntu - uv tools, hyperi-update tracks these) ansible.builtin.command: cmd: "uv tool install {{ item }}" environment: PATH: "{{ user_home }}/.cargo/bin:{{ user_home }}/.local/bin:{{ ansible_facts['env'].PATH }}" - become: false + become: true + become_user: "{{ actual_user }}" loop: - ruff - ty diff --git a/ansible/roles/developer-python/tasks/main.yml b/ansible/roles/developer-python/tasks/main.yml index 30be680..27ceb5b 100644 --- a/ansible/roles/developer-python/tasks/main.yml +++ b/ansible/roles/developer-python/tasks/main.yml @@ -15,7 +15,8 @@ cmd: uv tool install mypy environment: PATH: "{{ user_home }}/.cargo/bin:{{ user_home }}/.local/bin:{{ ansible_facts['env'].PATH }}" - become: false + become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" + become_user: "{{ actual_user }}" register: devpy_mypy_install changed_when: "'already installed' not in (devpy_mypy_install.stderr | default(''))" failed_when: false diff --git a/ansible/roles/developer/files/update/hyperi-update-linux.sh b/ansible/roles/developer/files/update/hyperi-update-linux.sh index 8c914ec..b0d5754 100644 --- a/ansible/roles/developer/files/update/hyperi-update-linux.sh +++ b/ansible/roles/developer/files/update/hyperi-update-linux.sh @@ -140,22 +140,53 @@ if [[ "$ASSUME_YES" -eq 0 ]]; then fi # --- sudo: ask once, keep alive ------------------------------------------- +# Probe with a real command, not `sudo -v`. Ubuntu 25.10+ replaced GNU sudo with +# sudo-rs, whose `-v` demands interactive authentication even where NOPASSWD +# grants the commands themselves — so `sudo -v` aborts this script on every +# passwordless box and under every unattended caller, which is exactly what +# --yes exists for. +# +# The keepalive only matters when a password was actually entered: NOPASSWD +# leaves no timestamp to refresh. section "Authenticating (sudo)" -if sudo -v; then +if sudo -n true 2>/dev/null; then + ok "sudo authenticated (passwordless)" +elif [[ -t 0 ]] && sudo -v; then ok "sudo authenticated" - # refresh the sudo timestamp in the background until the script exits ( while true; do sudo -n true 2>/dev/null; sleep 50; kill -0 "$$" 2>/dev/null || exit; done ) & SUDO_KEEPALIVE_PID=$! trap '[[ -n "${SUDO_KEEPALIVE_PID:-}" ]] && kill "$SUDO_KEEPALIVE_PID" 2>/dev/null' EXIT else printf '%s \xe2\x9c\x97 sudo authentication failed — aborting%s\n' "$RED" "$RESET" + printf '%s No passwordless sudo and no terminal to prompt on.%s\n' "$RED" "$RESET" exit 1 fi # --- System packages ------------------------------------------------------- +# unattended-upgrades and the apt-daily timers take the dpkg lock on their own +# schedule. Without this wait the upgrade exits 100 and the run reports success +# for everything else, so the box looks updated and is not. +wait_for_apt_lock() { + local waited=0 + while sudo fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1 || + sudo fuser /var/lib/apt/lists/lock >/dev/null 2>&1; do + if [[ "$waited" -eq 0 ]]; then + printf ' waiting for another apt process to finish...\n' + fi + if [[ "$waited" -ge 300 ]]; then + fail "apt lock (held for 5 minutes)" + return 1 + fi + sleep 5 + waited=$((waited + 5)) + done + return 0 +} + case "$PKG_MGR" in apt) section "APT — system packages" + wait_for_apt_lock run "apt-get update" sudo apt-get update run "apt-get full-upgrade" sudo apt-get -y full-upgrade run "apt-get autoremove" sudo apt-get -y autoremove diff --git a/ansible/roles/developer/tasks/nodejs.yml b/ansible/roles/developer/tasks/nodejs.yml index 933c021..ecd0305 100644 --- a/ansible/roles/developer/tasks/nodejs.yml +++ b/ansible/roles/developer/tasks/nodejs.yml @@ -167,7 +167,8 @@ - name: Install the previous Node major via fnm ansible.builtin.command: cmd: /usr/local/bin/fnm install {{ node_major_previous }} - become: false + become: true + become_user: "{{ actual_user }}" environment: FNM_DIR: "{{ user_home }}/.local/share/fnm" register: developer_fnm_install @@ -183,7 +184,8 @@ - name: Point the fnm default at the system Node ansible.builtin.command: cmd: /usr/local/bin/fnm default system - become: false + become: true + become_user: "{{ actual_user }}" environment: FNM_DIR: "{{ user_home }}/.local/share/fnm" register: developer_fnm_default @@ -288,10 +290,12 @@ become: false when: ansible_facts['distribution'] == 'MacOSX' +# become_user is ignored while become is false, so this has to escalate to +# reach the desktop user's ~/.npmrc rather than the connecting account's. - name: Configure the npm prefix (Linux) ansible.builtin.command: cmd: npm config set prefix "{{ user_home }}/.npm-global" - become: false + become: true become_user: "{{ actual_user }}" changed_when: false when: ansible_facts['distribution'] in ['Fedora', 'Ubuntu'] @@ -431,7 +435,7 @@ state: present environment: NPM_CONFIG_PREFIX: "{{ user_home }}/.npm-global" - become: false + become: true become_user: "{{ actual_user }}" when: ansible_facts['distribution'] in ['Fedora', 'Ubuntu'] diff --git a/ansible/roles/developer/tasks/removals.yml b/ansible/roles/developer/tasks/removals.yml index 845d441..0fd0204 100644 --- a/ansible/roles/developer/tasks/removals.yml +++ b/ansible/roles/developer/tasks/removals.yml @@ -56,6 +56,11 @@ # user-owned in ~/.local/bin there -- ~/.cargo/bin is a retired location on all # three, and it sits AHEAD of ~/.local/bin on PATH, so a copy left there # shadows the real install. +# +# Runs as the owner of that home, not as whoever connected. Under `become: +# false` on a machine where they differ, the file module cannot read the path +# and reports "Insufficient permissions ... Treating as absent" -- a green +# tombstone over a binary that is still there and still shadowing. - name: Remove the retired user-level uv/uvx (superseded by the current install) ansible.builtin.file: path: "{{ user_home }}/.cargo/bin/{{ item }}" @@ -63,7 +68,8 @@ loop: - uv - uvx - become: false + become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" + become_user: "{{ actual_user }}" # ============================================================================ # CANNOT CO-EXIST — the old thing actively conflicts with its replacement. @@ -137,6 +143,52 @@ when: ansible_facts['distribution'] == 'MacOSX' failed_when: false +# The package uninstall above does NOT take the docker context with it: context +# state is a per-user file under ~/.docker/contexts, owned by the user rather +# than the package. What is left points at a socket that no longer exists, and +# where `desktop-linux` is also the SELECTED context every docker command fails +# with "Cannot connect to the Docker daemon" until it is switched back. +# +# Context subcommands do not talk to the daemon, so these still work on a host +# whose selected context is already dead. +- name: Read the current docker context + ansible.builtin.command: + cmd: docker context show + become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" + become_user: "{{ actual_user }}" + register: developer_docker_ctx_current + changed_when: false + failed_when: false + +- name: List the docker contexts + ansible.builtin.command: + cmd: docker context ls --quiet + become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" + become_user: "{{ actual_user }}" + register: developer_docker_ctx_list + changed_when: false + failed_when: false + +# `docker context rm` refuses to remove the context in use, so this has to come +# first. +- name: Switch off the retired Docker Desktop context + ansible.builtin.command: + cmd: docker context use default + become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" + become_user: "{{ actual_user }}" + changed_when: true + failed_when: false + when: developer_docker_ctx_current.stdout | default('') | trim == 'desktop-linux' + +- name: Remove the retired Docker Desktop context + ansible.builtin.command: + cmd: docker context rm desktop-linux + become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" + become_user: "{{ actual_user }}" + changed_when: true + failed_when: false + when: "'desktop-linux' in (developer_docker_ctx_list.stdout_lines | default([]))" + # OnlyOffice - retired; LibreOffice is the org office suite now. Remove the # native package and the Flatpak fallback from hosts that carried it. - name: Remove OnlyOffice desktop editors (Fedora) diff --git a/ansible/roles/developer/tasks/verify.yml b/ansible/roles/developer/tasks/verify.yml index 5820bea..1ea4d9f 100644 --- a/ansible/roles/developer/tasks/verify.yml +++ b/ansible/roles/developer/tasks/verify.yml @@ -76,7 +76,7 @@ PATH: "/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:{{ user_home }}/.cargo/bin:{{ ansible_facts['env'].PATH }}" changed_when: false failed_when: verify_git_lfs.rc != 0 - become: false + become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" become_user: "{{ actual_user }}" - name: Verify yq diff --git a/ansible/roles/rdp-server/README.md b/ansible/roles/rdp-server/README.md index 79c0cb5..b4a1540 100644 --- a/ansible/roles/rdp-server/README.md +++ b/ansible/roles/rdp-server/README.md @@ -171,9 +171,84 @@ serving. This role restarts the service at the end of a run, so **applying it over RDP will drop your own connection mid-run**. Apply over SSH, from a local console, or accept the reconnect. +## Recovering from a failed handover + +Logging in over RDP while another session is already open for the same user +offers to force-stop it. Taking that offer makes gdm tear down the old session +**and the in-flight RDP login with it**, logged as: + + Gdm: GdmDisplay: Session never registered, failing + +The login screen itself is not the problem - it appears, and the password is +accepted. The new session is collateral damage from reaping the old one. + +What makes it feel permanent is the state it leaves behind: the system daemon +keeps accepting TCP on 3389 and services nothing. Every later connection gets a +blank screen, and the daemon does not log so much as an incoming connection. +Restarting `gdm` does **not** clear it - only restarting +`gnome-remote-desktop.service` does. On a headless host with no ssh, there is +no way back in at all. + +`rdp-handover-watchdog.service` watches gdm's journal for that message and +restarts the daemon, with a 60s cooldown so one failure cannot become a restart +loop. Disable with `rdp_handover_watchdog_enabled: false`. + +It matches the log rather than probing the port because telling a wedged daemon +from a healthy one over the wire needs a real RDP connection, and an aborted +one makes the daemon build and tear down a session - the same path that wedges +it. A periodic probe would risk causing the fault it watches for. + +Deployed only where `/usr/lib/systemd/user/gnome-remote-desktop-handover.service` +exists. That unit is the two-stage handover this fault lives in, so gating on it +tests the mechanism rather than a version number. + +### Testing it + +**The real path needs a LOCAL session**, not a second RDP one. Remote sessions +are kept alive and resumed, so RDP-into-RDP never offers to force-stop anything +and never reaches the fault. On a headless VM with no console user there is +nothing to force-stop either. + +At the machine's own keyboard: + +1. Log in locally and leave the session open. +2. RDP in from elsewhere as the same user. +3. Accept the offer to force-stop the other session. +4. That login dies either way - the bug is upstream. Reconnect: without the + watchdog every attempt is a blank screen forever; with it the reconnect + reaches a login screen. + + journalctl -u rdp-handover-watchdog -u gnome-remote-desktop --since -5m + +**Without the fault**, drive the watchdog off a synthetic unit -- the two +environment variables exist for this, and neither touches the real daemon or +any live session: + + sudo systemd-run --unit=wd-selftest \ + --setenv=RDP_WATCHDOG_SOURCE_UNIT=wd-selftest-source.service \ + --setenv=RDP_WATCHDOG_UNIT=wd-selftest-target.service \ + /usr/local/sbin/rdp-handover-watchdog + + sudo systemd-run --unit=wd-selftest-source \ + /usr/bin/echo "Gdm: GdmDisplay: Session never registered, failing" + + journalctl -u wd-selftest --no-pager + sudo systemctl stop wd-selftest + +Expect `failed handover detected -- restarting wd-selftest-target.service`. The +target does not exist, so the restart fails and says so; that is the point -- +it proves detection without touching RDP. + +### What it does not cover + +Only the gdm marker. A daemon wedged some other way is not detected, and the +force-stop still costs you that one login attempt - fixing that is upstream +gdm's problem, not this role's. + ## Files Modified - `/etc/gnome-remote-desktop/` - System certificates +- `/usr/local/sbin/rdp-handover-watchdog` + `/etc/systemd/system/rdp-handover-watchdog.service` - failed-handover recovery - `/etc/sysctl.d/98-rdp-tcp.conf` - TCP optimizations - `/etc/sysctl.d/98-rdp-mtu.conf` - MTU settings - `/etc/security/limits.d/50-rdp-nice.conf` - RLIMIT_NICE headroom for the handover daemon diff --git a/ansible/roles/rdp-server/defaults/main.yml b/ansible/roles/rdp-server/defaults/main.yml index 43638ee..e3a0233 100644 --- a/ansible/roles/rdp-server/defaults/main.yml +++ b/ansible/roles/rdp-server/defaults/main.yml @@ -42,3 +42,12 @@ rdp_cert_state: "NSW" rdp_cert_locality: "Sydney" rdp_cert_subject: >- /C={{ rdp_cert_country }}/ST={{ rdp_cert_state }}/L={{ rdp_cert_locality }}/O={{ rdp_cert_org }}/CN={{ ansible_hostname }} + +# Recover the RDP daemon after a failed handover. Force-stopping another +# session from the RDP login screen leaves the daemon accepting TCP on 3389 and +# servicing nothing, which on a headless host means no way back in at all. +# +# On by default: the failure is silent, and the only manual remedy needs the +# ssh access that a remote-desktop host may not have. Set false where something +# else already supervises the daemon. +rdp_handover_watchdog_enabled: true diff --git a/ansible/roles/rdp-server/files/rdp-handover-watchdog b/ansible/roles/rdp-server/files/rdp-handover-watchdog new file mode 100644 index 0000000..6876fe0 --- /dev/null +++ b/ansible/roles/rdp-server/files/rdp-handover-watchdog @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +# +# Restart GNOME Remote Desktop after a failed RDP handover. +# +# Force-stopping another session from the RDP login screen makes gdm tear down +# that session and the in-flight RDP login with it, logged as "GdmDisplay: +# Session never registered, failing". The system daemon is then left accepting +# TCP on 3389 and servicing nothing -- later connections get a blank screen and +# no log line, and only a restart clears it. +# +# Matched from the log rather than probed: telling wedged from healthy over the +# wire needs an RDP connection, and an aborted one exercises the same session +# teardown that wedges the daemon. +# +# Restarting drops any live RDP session, which is acceptable only because the +# trigger means a login has already failed. + +set -euo pipefail + +UNIT="${RDP_WATCHDOG_UNIT:-gnome-remote-desktop.service}" +SOURCE_UNIT="${RDP_WATCHDOG_SOURCE_UNIT:-gdm.service}" +MARKER='Session never registered, failing' + +# A single failure can log the marker more than once, and a restart must not +# become a loop. +COOLDOWN="${RDP_WATCHDOG_COOLDOWN:-60}" + +# systemd already routes stderr to the journal under this unit, so `logger` +# would file a second copy of every line. +log() { printf '%s\n' "$1" >&2; } + +log "watching ${SOURCE_UNIT} for a failed handover" + +last_restart=0 + +# Process substitution, not a pipe: the last stage of a pipeline runs in a +# subshell, where last_restart would reset on every line and defeat the +# cooldown. +while IFS= read -r line; do + case "$line" in + *"$MARKER"*) ;; + *) continue ;; + esac + + now="$(date +%s)" + if (( now - last_restart < COOLDOWN )); then + log "failed handover again within ${COOLDOWN}s -- not restarting" + continue + fi + last_restart="$now" + + log "failed handover detected -- restarting ${UNIT}" + if systemctl restart "$UNIT"; then + log "${UNIT} restarted; reconnect should now reach a login screen" + else + log "ERROR: failed to restart ${UNIT}" + fi +done < <(journalctl --follow --lines 0 --output cat --unit "$SOURCE_UNIT") diff --git a/ansible/roles/rdp-server/files/rdp-handover-watchdog.service b/ansible/roles/rdp-server/files/rdp-handover-watchdog.service new file mode 100644 index 0000000..1e9370f --- /dev/null +++ b/ansible/roles/rdp-server/files/rdp-handover-watchdog.service @@ -0,0 +1,19 @@ +[Unit] +Description=Restart GNOME Remote Desktop after a failed RDP handover +Documentation=file:/usr/local/sbin/rdp-handover-watchdog +After=gdm.service gnome-remote-desktop.service +Wants=gnome-remote-desktop.service + +[Service] +Type=simple +ExecStart=/usr/local/sbin/rdp-handover-watchdog +# The watchdog exits if journalctl --follow ever dies; without this the host +# silently loses its only route back in after a failed handover. +Restart=always +RestartSec=5 +ProtectHome=yes +PrivateTmp=yes +NoNewPrivileges=yes + +[Install] +WantedBy=multi-user.target diff --git a/ansible/roles/rdp-server/tasks/handover_watchdog.yml b/ansible/roles/rdp-server/tasks/handover_watchdog.yml new file mode 100644 index 0000000..8d0b90b --- /dev/null +++ b/ansible/roles/rdp-server/tasks/handover_watchdog.yml @@ -0,0 +1,83 @@ +--- +# Watchdog for the wedged-daemon fault (the why is in +# files/rdp-handover-watchdog). + +# The two-stage handover, and the fault, arrived with the GNOME 46 remote-login +# rework. Gating on the unit rather than a version number tests the mechanism +# itself, so it holds across distros and across a backport. +- name: Stat the GRD user handover unit + ansible.builtin.stat: + path: /usr/lib/systemd/user/gnome-remote-desktop-handover.service + register: rdp_handover_unit + when: + - ansible_facts['distribution'] in ['Fedora', 'Ubuntu'] + - has_gnome | default(false) + +- name: Install the handover watchdog + when: + - ansible_facts['distribution'] in ['Fedora', 'Ubuntu'] + - has_gnome | default(false) + - rdp_handover_watchdog_enabled | bool + - rdp_handover_unit.stat.exists | default(false) + block: + - name: Deploy the handover watchdog + ansible.builtin.copy: + src: rdp-handover-watchdog + dest: /usr/local/sbin/rdp-handover-watchdog + mode: '0755' + owner: root + group: root + register: rdp_watchdog_script + + - name: Install the handover watchdog service + ansible.builtin.copy: + src: rdp-handover-watchdog.service + dest: /etc/systemd/system/rdp-handover-watchdog.service + mode: '0644' + owner: root + group: root + register: rdp_watchdog_unit_file + + # Restarted rather than started when either file moved: a watchdog still + # running the previous script is not watching what was just deployed. + - name: Enable and start the handover watchdog + ansible.builtin.systemd_service: + name: rdp-handover-watchdog.service + enabled: true + state: >- + {{ 'restarted' + if (rdp_watchdog_script.changed or rdp_watchdog_unit_file.changed) + else 'started' }} + daemon_reload: true + when: not ansible_check_mode + +# Turning the watchdog off has to take the running unit with it, or it keeps +# restarting the daemon from a script nobody maintains any more. +- name: Remove the handover watchdog + when: + - ansible_facts['distribution'] in ['Fedora', 'Ubuntu'] + - not (rdp_handover_watchdog_enabled | bool) + block: + - name: Stop and disable the handover watchdog + ansible.builtin.systemd_service: + name: rdp-handover-watchdog.service + enabled: false + state: stopped + failed_when: false + when: not ansible_check_mode + + - name: Remove the handover watchdog files + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: + - /etc/systemd/system/rdp-handover-watchdog.service + - /usr/local/sbin/rdp-handover-watchdog + register: rdp_watchdog_removed + + - name: Reload systemd after removing the watchdog + ansible.builtin.systemd_service: + daemon_reload: true + when: + - rdp_watchdog_removed.changed + - not ansible_check_mode diff --git a/ansible/roles/rdp-server/tasks/main.yml b/ansible/roles/rdp-server/tasks/main.yml index f155d18..376a69e 100644 --- a/ansible/roles/rdp-server/tasks/main.yml +++ b/ansible/roles/rdp-server/tasks/main.yml @@ -73,6 +73,16 @@ file: pam_nice.yml tags: ['pam_nice'] +# `apply:` as well as the include's own tag -- without it a `--tags +# handover-watchdog` run includes the file and then filters out every task in +# it. +- name: Recover the RDP daemon after a failed handover + ansible.builtin.include_tasks: + file: handover_watchdog.yml + apply: + tags: ['handover-watchdog'] + tags: ['handover-watchdog'] + - name: Verify RDP optimizations ansible.builtin.include_tasks: file: verify.yml diff --git a/ansible/roles/soe/tasks/claude.yml b/ansible/roles/soe/tasks/claude.yml index ebfc735..1b3bbd4 100644 --- a/ansible/roles/soe/tasks/claude.yml +++ b/ansible/roles/soe/tasks/claude.yml @@ -27,7 +27,7 @@ state: absent environment: NPM_CONFIG_PREFIX: "{{ user_home }}/.npm-global" - become: false + become: true become_user: "{{ actual_user }}" failed_when: false when: ansible_facts['distribution'] in ['Fedora', 'Ubuntu'] diff --git a/ansible/roles/soe/tasks/verify.yml b/ansible/roles/soe/tasks/verify.yml index db96030..1cc26b2 100644 --- a/ansible/roles/soe/tasks/verify.yml +++ b/ansible/roles/soe/tasks/verify.yml @@ -24,12 +24,16 @@ failed_when: verify_openvpn.rc != 0 when: ansible_facts['distribution'] == 'MacOSX' +# Run as the user whose home this is, not as whoever connected. `become: false` +# means the connection user, and on a fleet machine that is a service account +# (hyperi-infra connects as `ubuntu`) with no access to the desktop user's home. - name: Verify Claude Code CLI (Linux) ansible.builtin.command: "{{ user_home }}/.local/bin/claude --version" register: verify_claude changed_when: false failed_when: verify_claude.rc != 0 - become: false + become: true + become_user: "{{ actual_user }}" when: ansible_facts['distribution'] in ['Fedora', 'Ubuntu'] - name: Verify Claude Code CLI (macOS) diff --git a/docs/install-matrix.md b/docs/install-matrix.md index 648d350..f1bf6e9 100644 --- a/docs/install-matrix.md +++ b/docs/install-matrix.md @@ -271,7 +271,7 @@ hyperi-ci. | Role | Tag | What it does | |---|---|---| | `power-profile` | `power-profile` | Sleep, idle and lid policy. `always-on` (default) never sleeps on mains power, lid shut included, and leaves battery behaviour stock; `vm` never sleeps at all, for an unattended RDP guest. Select with `-e power_profile=`. Profiles are data files under `roles/power-profile/vars/profiles/` | -| `rdp-server` | `rdp-server` | GNOME Remote Desktop on port 3389 | +| `rdp-server` | `rdp-server` | GNOME Remote Desktop on port 3389. Includes `handover-watchdog`, which restarts the daemon after a failed handover leaves it accepting connections and servicing none | | `vm` / `optimizer` | `vm` | VM guest optimisations (QEMU/SPICE agents) | Deliberately outside `soe`: the right power answer differs per machine, and