Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ Fleet guards (`fleet/`) are runtime scripts deployed to `~/oss-fleet/` on the bo
1. Write the script in `fleet/doctor/` or `fleet/browsers/`.
2. Add a `.j2` unit template in `ansible/templates/`.
3. Wire it in `ansible/tasks/fleet_guards.yml` (or `fleet-browsers.yml`).
4. Add it to `factory_fleet_units` and/or `factory_fleet_enabled_units` in `group_vars/all.yml`.
4. Add it to `factory_fleet_units` and/or `factory_fleet_enabled_units` (or `factory_fleet_browser_units` / `factory_fleet_browser_enabled_units`) in `group_vars/all.yml`.
5. Update `docs/fleet-guards.md`.

## Pull requests
Expand Down
26 changes: 17 additions & 9 deletions ansible/group_vars/all.yml
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,8 @@ factory_installer_also: >-
{{ (['koncreet'] if (factory_cfg.start_services | bool) else [])
+ (['psutil'] if ((factory_cfg.profiles.agents | bool)
and (factory_cfg.browser_prune.enabled | bool)) else [])
+ (['obscura', 'supabase'] if (factory_cfg.profiles.fleet_guards | bool) else []) }}
+ (['obscura'] if (factory_fleet_browsers_enabled | bool) else [])
+ (['supabase'] if (factory_cfg.profiles.fleet_guards | bool) else []) }}

# --- Herdr static config ----------------------------------------------------
# Only the reviewed, non-secret subset is templated. experimental.pane_history
Expand Down Expand Up @@ -328,11 +329,26 @@ factory_shared_supabase_db_volume: "supabase_db_{{ factory_shared_supabase_proje
factory_shared_supabase_storage_volume: "supabase_storage_{{ factory_shared_supabase_project }}"
# The npm registry's latest supabase, resolved once per run (factory_latest).
factory_shared_supabase_cli_version: "{{ factory_latest.supabase }}"

# --- Fleet browsers (profiles.fleet_browsers, or fleet_guards) ---------------
factory_fleet_browsers_enabled: "{{ (factory_cfg.profiles.fleet_guards | bool) or (factory_cfg.profiles.fleet_browsers | bool) }}"
factory_fleet_browsers_dir: "{{ factory_fleet_dir }}/browsers"
# The latest Obscura release for this platform, with the SHA-256 GitHub
# publishes for the asset; each release extracts into its own directory.
factory_browser_obscura: "{{ factory_latest.obscura.assets[factory_platform] }}"
factory_browser_obscura_dir: "{{ factory_fleet_browsers_dir }}/obscura-{{ factory_latest.obscura.version }}"
factory_fleet_browser_units:
- fleet-browser-obscura.service
- fleet-browser-chrome.service
- fleet-browser-vnc.service
- fleet-browser-sync.service
- fleet-browser-sync.timer
- fleet-browser-gc.service
- fleet-browser-gc.timer
factory_fleet_browser_enabled_units:
- { unit: fleet-browser-obscura.service, wants: default.target.wants }
- { unit: fleet-browser-sync.timer, wants: timers.target.wants }
- { unit: fleet-browser-gc.timer, wants: timers.target.wants }

# Storage guard (fleet/doctor/storage-guard.sh, every 5 min). Use% of the
# filesystems holding /, /var/log, the home and Docker's data root: WARN alerts
Expand Down Expand Up @@ -361,11 +377,6 @@ factory_fleet_units:
- flotilla-storage-guard.timer
- flotilla-devtools-bridge-reaper.service
- flotilla-devtools-bridge-reaper.timer
- fleet-browser-obscura.service
- fleet-browser-sync.service
- fleet-browser-sync.timer
- fleet-browser-gc.service
- fleet-browser-gc.timer
factory_fleet_enabled_units:
- { unit: flotilla-shared-supabase.service, wants: default.target.wants }
- { unit: flotilla-docker-guard.service, wants: default.target.wants }
Expand All @@ -375,9 +386,6 @@ factory_fleet_enabled_units:
- { unit: flotilla-dev-server-reaper.timer, wants: timers.target.wants }
- { unit: flotilla-storage-guard.timer, wants: timers.target.wants }
- { unit: flotilla-devtools-bridge-reaper.timer, wants: timers.target.wants }
- { unit: fleet-browser-obscura.service, wants: default.target.wants }
- { unit: fleet-browser-sync.timer, wants: timers.target.wants }
- { unit: fleet-browser-gc.timer, wants: timers.target.wants }

# --- Tailscale (install only; login, ACLs, SSH and UFW stay manual) ---------
factory_tailscale_track: stable
Expand Down
4 changes: 2 additions & 2 deletions ansible/site.yml
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,8 @@
ansible.builtin.import_tasks: tasks/fleet-browsers.yml
when:
- factory_account_ready | bool
- factory_cfg.profiles.fleet_guards | bool
tags: [fleet]
- factory_fleet_browsers_enabled | bool
tags: [fleet, fleet_browsers]

- name: Tailscale package installation only
ansible.builtin.import_tasks: tasks/tailscale.yml
Expand Down
45 changes: 13 additions & 32 deletions ansible/tasks/fleet-browsers.yml
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
---
# Fleet browsers: obscura -> chrome -> vnc tier ladder, shared session jar.
# profiles.fleet_browsers or profiles.fleet_guards turns it on.
# docs/fleet-guards.md owns the design; fleet/browsers/fleet-browser owns the
# runtime. Only the obscura tier needs provisioning (latest binary + unit);
# chrome and vnc are started on demand by the operator or the fleet-browser
# CLI and gc'd after idle_timeout_min.
# CLI and gc'd after idle_timeout_min. The Herdr unit carries the ladder's
# environment (templates/herdr.service.j2), so this file never edits it.

- name: Ensure the journald drop-in directory
tags: [journald]
ansible.builtin.file:
path: /etc/systemd/journald.conf.d
state: directory
Expand All @@ -20,6 +23,7 @@
# lines per 30s (scaled up by journald with free disk). A page stuck in a loop
# once logged ~17 MB/s and filled the disk via syslog.
- name: Cap journald intake host-wide
tags: [journald]
ansible.builtin.copy:
dest: /etc/systemd/journald.conf.d/50-fleet-ratelimit.conf
content: |
Expand All @@ -42,8 +46,11 @@
mode: "0755"
become: true
loop:
- "{{ factory_fleet_dir }}"
- "{{ factory_fleet_browsers_dir }}"
- "{{ factory_cfg.home }}/.fleet-browser"
- "{{ factory_user_units }}/default.target.wants"
- "{{ factory_user_units }}/timers.target.wants"

- name: Install the fleet browser scripts
ansible.builtin.copy:
Expand Down Expand Up @@ -132,21 +139,6 @@
- .profile
- .bashrc

- name: Inject the shared-browser env into herdr.service
ansible.builtin.lineinfile:
path: "{{ factory_cfg.home }}/.config/systemd/user/herdr.service"
insertafter: "^\\[Service\\]"
line: "{{ item }}"
owner: "{{ factory_cfg.user }}"
group: "{{ factory_group }}"
mode: "0644"
become: true
loop:
- "Environment=CHROME_DEVTOOLS_AXI_BROWSER_URL=http://127.0.0.1:9222"
- "Environment=FLEET_BROWSER_TIER=obscura"
- "Environment=PATH={{ factory_fleet_browsers_dir }}:{{ factory_user_env.PATH }}"
notify: reload user systemd

- name: Render fleet browser units
ansible.builtin.template:
src: "{{ item }}.j2"
Expand All @@ -155,14 +147,7 @@
group: "{{ factory_group }}"
mode: "0644"
become: true
loop:
- fleet-browser-obscura.service
- fleet-browser-chrome.service
- fleet-browser-vnc.service
- fleet-browser-sync.service
- fleet-browser-sync.timer
- fleet-browser-gc.service
- fleet-browser-gc.timer
loop: "{{ factory_fleet_browser_units }}"
notify:
- reload user systemd
- restart fleet browser obscura
Expand All @@ -178,10 +163,9 @@
force: true
follow: false
become: true
loop:
- { unit: fleet-browser-obscura.service, wants: default.target.wants }
- { unit: fleet-browser-sync.timer, wants: timers.target.wants }
- { unit: fleet-browser-gc.timer, wants: timers.target.wants }
loop: "{{ factory_fleet_browser_enabled_units }}"
loop_control:
label: "{{ item.unit }}"

- name: Enable and start fleet browser units
ansible.builtin.systemd_service:
Expand All @@ -192,10 +176,7 @@
become: "{{ factory_become_target | bool }}"
become_user: "{{ factory_cfg.user }}"
environment: "{{ factory_user_systemd_env }}"
loop:
- fleet-browser-obscura.service
- fleet-browser-sync.timer
- fleet-browser-gc.timer
loop: "{{ factory_fleet_browser_enabled_units | map(attribute='unit') | list }}"
when: factory_manage_services | bool

- name: Report the deferred fleet browser start
Expand Down
14 changes: 13 additions & 1 deletion ansible/tasks/verify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,19 @@
| map('join', '/')
| map('regex_replace', '^', factory_user_units ~ '/')
| list)
if (factory_cfg.profiles.fleet_guards | bool) else []) }}
if (factory_cfg.profiles.fleet_guards | bool) else [])
+ ([factory_fleet_browsers_dir ~ '/fleet-browser',
factory_fleet_browsers_dir ~ '/cookie-sync.ts',
factory_fleet_browsers_dir ~ '/env.sh']
+ (factory_fleet_browser_units
| map('regex_replace', '^', factory_user_units ~ '/')
| list)
+ (factory_fleet_browser_enabled_units | map(attribute='wants')
| zip(factory_fleet_browser_enabled_units | map(attribute='unit'))
| map('join', '/')
| map('regex_replace', '^', factory_user_units ~ '/')
| list)
if (factory_fleet_browsers_enabled | bool) else []) }}

- name: Stat every expected output
ansible.builtin.stat:
Expand Down
7 changes: 6 additions & 1 deletion ansible/templates/herdr.service.j2
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,12 @@ Documentation=https://herdr.dev/docs/
Type=simple
ExecStart={{ factory_herdr_bin }} server
WorkingDirectory={{ factory_cfg.workspace }}
Environment=PATH=%h/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Environment=PATH={{ (factory_fleet_browsers_dir ~ ':') if (factory_fleet_browsers_enabled | bool) else '' }}%h/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
{% if factory_fleet_browsers_enabled | bool %}
# The fleet browser ladder's default tier (docs/fleet-guards.md#browser-ladder).
Environment=CHROME_DEVTOOLS_AXI_BROWSER_URL=http://127.0.0.1:9222
Environment=FLEET_BROWSER_TIER=obscura
{% endif %}
{% if factory_cfg.profiles.agents | bool %}
# The installed MCP entrypoint, so AXI never resolves a floating npx package.
{% for key, value in factory_managed_shell_env | dictsort %}
Expand Down
1 change: 1 addition & 0 deletions config/default.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ factory:
tailscale: false
desktop: false
fleet_guards: false
fleet_browsers: false
herdr:
theme: catppuccin
toast_delivery: herdr
Expand Down
2 changes: 1 addition & 1 deletion docs/agent-host-move.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Services that hold state other hosts share, such as a model relay, a monitoring

## 1. Provision the new host

1. Follow the [Quick start](../README.md#quick-start): bootstrap, init, validate, plan, apply, doctor. In `.local/host.yml`, enable the same profiles as the old host, plus `desktop`: the fleet-browser sign-in in [What git does not carry](#2-what-git-does-not-carry) needs its TigerVNC and noVNC packages, and with `desktop` on the default installs `/usr/bin/google-chrome`, which the fleet-browser finds first. The first apply installs omp; until omp has a provider login it skips the questions and prints a line saying to sign in.
1. Follow the [Quick start](../README.md#quick-start): bootstrap, init, validate, plan, apply, doctor. In `.local/host.yml`, enable the same profiles as the old host, plus `fleet_browsers` and `desktop`: the fleet-browser sign-in in [What git does not carry](#2-what-git-does-not-carry) needs the browser ladder and the `desktop` TigerVNC and noVNC packages, and with `desktop` on the default installs `/usr/bin/google-chrome`, which the fleet-browser finds first. The first apply installs omp; until omp has a provider login it skips the questions and prints a line saying to sign in.
2. Sign in to omp ([Sign in](omp.md#sign-in)), then rerun `./factory apply` in an interactive terminal. It opens Firstmate on omp, which asks the move decisions one question at a time. Later applies skip the questions; to open them again, delete `~/.local/share/code-factory/new-host-questions-done` and rerun `./factory apply` interactively.
3. Fetch `~/super.env` as described in [Fetch on a new host](secrets.md#fetch-on-a-new-host). Compare its `sha256sum` with the old host's copy.
4. Join the tailnet as a new device ([Remote access](security.md#remote-access)).
Expand Down
2 changes: 1 addition & 1 deletion docs/capacity.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ Idle chrome-devtools-axi bridges come on top: each holds about 2 GB until
| Pruner | Profile | Runs | Removes |
| --- | --- | --- | --- |
| `chrome-autoprune.timer` → `chrome-autoprune.py --apply` | `agents` (`browser_prune.enabled`) | every `poll_seconds` (300 s) | AXI bridge browser processes idle past `idle_seconds` (7200 s). It never touches headed, attached or persistent-profile browsers. |
| `fleet-browser-gc.timer` | `fleet_guards` | every 5 min | Stops browser ladder tiers 2 and 3 (`chrome`, `vnc`) after 30 idle minutes with no CDP client (`FLEET_BROWSER_IDLE_MIN`). |
| `fleet-browser-gc.timer` | `fleet_browsers` or `fleet_guards` | every 5 min | Stops browser ladder tiers 2 and 3 (`chrome`, `vnc`) after 30 idle minutes with no CDP client (`FLEET_BROWSER_IDLE_MIN`). |
| `flotilla-dev-server-reaper.timer` → `dev-server-reaper.sh` | `fleet_guards` | every 2 min | `next dev` / `next-server` / `tsc --noEmit` trees whose lane is done, paused, blocked or failed, has no agent, or has been idle 30 min or more (`REAPER_IDLE_MIN`). |
| `flotilla-storage-guard.timer` → `storage-guard.sh` | `fleet_guards` | every 5 min | At CRIT (92%): build cache, dangling images, unused images older than `factory_storage_guard_image_age_hours`. Never volumes, containers, repositories, logs or home content. |
| `flotilla-devtools-bridge-reaper.timer` → `devtools-bridge-reaper.sh` | `fleet_guards` | every 10 min | Attached chrome-devtools-axi bridges (`CHROME_DEVTOOLS_AXI_BROWSER_URL` set) whose process tree used no CPU and whose session state files did not change for 60 min (`REAPER_IDLE_MIN`); never any other bridge. |
Expand Down
6 changes: 4 additions & 2 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@ factory:
docker: true # Docker engine (group membership opt-in separately)
tailscale: false # Daemon only; authenticate separately
desktop: false # XFCE + TigerVNC + noVNC
fleet_guards: false # Shared Supabase, browser ladder
fleet_guards: false # Shared Supabase and the fleet's guards
fleet_browsers: false # Browser ladder (obscura tier on 127.0.0.1:9222)
herdr:
theme: catppuccin
sidebar_width: 46 # Spaces and Agents sidebar layouts: see herdr.md
Expand Down Expand Up @@ -79,7 +80,8 @@ The recipe refuses to overwrite a conflicting unmanaged command or an independen
| `development` | on | Rust toolchain (stable), build essentials. |
| `firstmate` | on | Firstmate clone tracking upstream `main`, plus seeded Firstmate config: crew dispatch, crew and secondmate harness, the crew omp overlay (crew advisor, see [omp configuration](omp.md#advisor)), Herdr backend selection, startup memory budget, and the spawn memory floor. |
| `docker` | on | Docker engine and Compose v2, with daemon defaults `init` (reaps orphaned children) and `live-restore`. Group membership is opt-in through the Ansible variable `factory_docker_group_users`. |
| `fleet_guards` | off | Shared Supabase stack, Docker event guard, [browser ladder](fleet-guards.md#browser-ladder), dev-server reaper, devtools-bridge reaper, storage guard, env seeder. See [Fleet guards](fleet-guards.md). |
| `fleet_guards` | off | Shared Supabase stack, Docker event guard, dev-server reaper, devtools-bridge reaper, storage guard, env seeder. See [Fleet guards](fleet-guards.md). |
| `fleet_browsers` | off | The [browser ladder](fleet-guards.md#browser-ladder): the always-on Obscura CDP tier on `127.0.0.1:9222`, the on-demand `chrome` and `vnc` tiers, the cookie sync and gc timers, and the ladder environment in shell profiles and the Herdr unit. `fleet_guards` provisions the same ladder, so a `fleet_guards` host needs no change. Needs no other profile; its `vnc` tier needs the `desktop` packages, and the ladder needs `iproute2` (`ss`) from the base image, which only `desktop` installs. |
| `tailscale` | off | Tailscale daemon only. Authentication is manual; see [Security](security.md#remote-access). |
| `desktop` | off | Loopback-only XFCE + TigerVNC + noVNC operator desktop on `127.0.0.1:6080`, and the Google Chrome apt package. Needs an operator-created VNC password; see [Desktop access](recovery.md#desktop-access). Also supplies the TigerVNC/noVNC packages the browser ladder's `vnc` tier needs. |

Expand Down
8 changes: 3 additions & 5 deletions docs/dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,8 @@ The GitHub lookups use the GitHub API, which allows 60 unauthenticated requests

## Fleet browsers and Supabase

`fleet_guards` profile.

- Obscura, the latest [h4ckf0r0day/obscura release](https://github.com/h4ckf0r0day/obscura/releases/latest) for the host's platform, verified against the GitHub release-asset digest (`ansible/tasks/fleet-browsers.yml`). Each release extracts into its own `~/oss-fleet/browsers/obscura-<version>/`.
- Supabase CLI, the npm registry's latest `supabase`, installed with `npm install` into `~/oss-fleet/shared-supabase` (`ansible/tasks/fleet_guards.yml`).
- `fleet_browsers` or `fleet_guards` profile: Obscura, the latest [h4ckf0r0day/obscura release](https://github.com/h4ckf0r0day/obscura/releases/latest) for the host's platform, verified against the GitHub release-asset digest (`ansible/tasks/fleet-browsers.yml`). Each release extracts into its own `~/oss-fleet/browsers/obscura-<version>/`.
- `fleet_guards` profile: Supabase CLI, the npm registry's latest `supabase`, installed with `npm install` into `~/oss-fleet/shared-supabase` (`ansible/tasks/fleet_guards.yml`).

## Koncreet

Expand Down Expand Up @@ -80,5 +78,5 @@ Also needed, depending on profile:

- Membership in the `docker` group for the account that runs fleet guards. Opt in with `factory_docker_group_users`.
- `psmisc` (`fuser`) for `fleet-browser seed`.
- `iproute2` (`ss`) for the CLIENTS column of `fleet-browser status`. Without `ss`, every tier reports 0 clients and gc can stop a tier in use. Only the `desktop` profile installs `iproute2`.
- `iproute2` (`ss`) for the CLIENTS column of `fleet-browser status`. Without `ss`, every tier reports 0 clients and gc can stop a tier in use. Only the `desktop` profile installs `iproute2`, so a `fleet_browsers` or `fleet_guards` host without `desktop` needs it in the base image.
- A VNC password created by the operator, for the `desktop` profile.
10 changes: 8 additions & 2 deletions docs/fleet-guards.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,12 +99,18 @@ stubbed `df`, `du` and `docker`.

## Browser ladder

Three browser tiers share one cookie jar. The `fleet_guards` profile installs
them under `~/oss-fleet/browsers/`. Sources of truth:
Three browser tiers share one cookie jar. The `fleet_browsers` profile, or
`fleet_guards`, installs them under `~/oss-fleet/browsers/`, and puts their
environment in shell profiles and the Herdr unit. Sources of truth:
`fleet/browsers/fleet-browser` (runtime, `alive` probe),
`fleet/browsers/env.sh` (defaults every shell inherits),
`fleet/browsers/cookie-sync.ts` (jar), `ansible/tasks/fleet-browsers.yml` and
`ansible/templates/fleet-browser-*.{service,timer}.j2` (units, cadences).
The playbook tag `fleet_browsers` (or `fleet`) narrows a run to the ladder;
`--skip-tags journald` also leaves out its host-wide journald cap.
Those runs leave the Herdr unit alone. The unit gets the ladder environment
only on a run that includes the `herdr` tag, for example
`--tags herdr,fleet_browsers`, and that run restarts Herdr.

| Tier | CDP port | Always on? | Use it when |
| --- | --- | --- | --- |
Expand Down
2 changes: 1 addition & 1 deletion fleet/browsers/fleet-browser
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ HERE=$(cd "$(dirname "$0")" && pwd)
STATE=$HOME/.fleet-browser
LOG=$STATE/fleet-browser.log
IDLE_MIN=${FLEET_BROWSER_IDLE_MIN:-30}
BUN=${BUN:-$HOME/.bun/bin/bun}
BUN=${BUN:-bun}
OBSCURA_URL=http://127.0.0.1:9222
CHROME_URL=http://127.0.0.1:9522
VNC_URL=http://127.0.0.1:9523
Expand Down
3 changes: 3 additions & 0 deletions schemas/factory.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,9 @@
},
"fleet_guards": {
"type": "boolean"
},
"fleet_browsers": {
"type": "boolean"
}
},
"required": [
Expand Down
Loading