Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .agents/skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,31 @@ For source checkout development, restart the local gateway with:
mise run gateway:docker
```

For the experimental host-supervised Firecracker path, use:

```bash
mise run gateway:firecracker
```

This task requires Linux, read/write `/dev/kvm`, `debugfs`, and the Firecracker,
kernel, and ext4 fixtures documented under `e2e/firecracker/`. It starts an
operator-managed `firecracker` compute-driver socket and a plaintext gateway.
The driver log defaults below `/tmp/openshell-firecracker-<user>-<gateway>/`.
The prototype creates no TAP device or guest NIC and needs neither `sudo` nor
`CAP_NET_ADMIN`.

If an older checkout fails while compiling `z3-sys` or another bindgen consumer
with `fatal error: 'stdbool.h' file not found`, set the GCC architecture header
path before retrying:

```bash
export BINDGEN_EXTRA_CLANG_ARGS="${BINDGEN_EXTRA_CLANG_ARGS:+${BINDGEN_EXTRA_CLANG_ARGS} }-isystem $(gcc -print-file-name=include)"
mise run gateway:firecracker
```

The current Firecracker gateway task applies this fallback automatically after
its optional `sg kvm` re-exec.

### Step 5: Check Podman-Backed Gateways

```bash
Expand Down Expand Up @@ -429,6 +454,14 @@ Use the VM driver logs and host diagnostics available in the user's environment.
- Host virtualization support is enabled.
- The sandbox supervisor can establish its callback connection to the gateway.

For Firecracker, also verify the configured external socket and driver log:

```bash
rg -n 'firecracker|socket_path' .cache/gateway-firecracker/gateway.toml
stat /tmp/openshell-firecracker-*/compute-driver.sock
tail -n 200 /tmp/openshell-firecracker-*/driver.log
```

Then run:

```bash
Expand Down
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ These pipelines connect skills into end-to-end workflows. Individual skill files
| `crates/openshell-driver-docker/` | Docker compute driver | In-process `ComputeDriver` backend for local Docker sandbox containers |
| `crates/openshell-driver-podman/` | Podman compute driver | In-process `ComputeDriver` backend for local Podman sandbox containers |
| `crates/openshell-driver-vm/` | VM compute driver | Standalone libkrun-backed `ComputeDriver` subprocess (embeds its own rootfs + runtime) |
| `crates/openshell-driver-firecracker/` | Firecracker isolation driver | Experimental host-side RFC 0012 backend with a private guest process-supervisor leaf mode |
| `crates/openshell-prover/` | Policy prover | Policy verification and proof generation |
| `crates/openshell-server-macros/` | Server macros | Compile-time helpers for gateway RPC authorization |
| `crates/openshell-supervisor-middleware/` | Middleware runtime | Generic middleware registry, remote service integration, and chain execution |
Expand Down
26 changes: 26 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, and CDI GPU devices when available. |
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
| Firecracker | Experimental host-supervised microVM isolation. | Per-sandbox no-NIC Firecracker VM with the logical supervisor on the host. | Operator-managed endpoint driver started by `mise run gateway:firecracker`. The prototype clones a configured ext4 fixture, injects the private guest mode, and passes an authenticated RFC 0012 topology descriptor to the host supervisor. OCI image materialization and the full exec/forwarding surfaces remain follow-up work. |
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |

Per-sandbox CPU and memory values currently enter the driver layer through
Expand Down Expand Up @@ -158,6 +159,7 @@ Runtime-specific implementation notes belong in the driver crate README:
- `crates/openshell-driver-podman/README.md`
- `crates/openshell-driver-kubernetes/README.md`
- `crates/openshell-driver-vm/README.md`
- `crates/openshell-driver-firecracker/README.md`

The combined VM topology runs `openshell-sandbox` as guest PID 1. libkrun
executes the driver-owned guest bootstrap as PID 1, and the bootstrap preserves
Expand Down
8 changes: 8 additions & 0 deletions architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,14 @@ by default. The co-located backend requires the
supervisor to own the execution environment's PID namespace so boundary
teardown can terminate every remaining workload process.

The experimental Firecracker backend exercises a delegated placement. The
logical supervisor stays on the host and sends the admitted policy and workload
spec over an authenticated, backend-private virtio-vsock channel only after the
RFC lifecycle reaches `start_agent`. A private guest mode of the Firecracker
driver invokes the existing process-supervisor implementation as the in-VM
leaf. The prototype attaches no guest NIC, so network access is blocked by the
VM topology while mediated egress remains unimplemented.

For proxy-mode boundaries, the co-located backend verifies its default-deny
kernel egress ceiling before exposing any workload execution surface and then
rechecks it every 250 milliseconds. Each check has a two-second deadline.
Expand Down
46 changes: 46 additions & 0 deletions crates/openshell-driver-firecracker/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

[package]
name = "openshell-driver-firecracker"
description = "Experimental Firecracker isolation driver for OpenShell"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
repository.workspace = true

[lib]
name = "openshell_driver_firecracker"
path = "src/lib.rs"

[[bin]]
name = "openshell-driver-firecracker"
path = "src/main.rs"

[dependencies]
openshell-core = { path = "../openshell-core", default-features = false }
openshell-isolation = { path = "../openshell-isolation" }
openshell-supervisor-process = { path = "../openshell-supervisor-process" }

async-trait = "0.1"
base64 = { workspace = true }
clap = { workspace = true }
futures = { workspace = true }
libc = "0.2"
nix = { workspace = true }
rand = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
thiserror = { workspace = true }
tokio = { workspace = true }
tokio-stream = { workspace = true, features = ["net"] }
tonic = { workspace = true, features = ["transport"] }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }

[dev-dependencies]
tempfile = "3"

[lints]
workspace = true
56 changes: 56 additions & 0 deletions crates/openshell-driver-firecracker/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# OpenShell Firecracker driver

`openshell-driver-firecracker` is an experimental host-side implementation of
the RFC 0012 Isolation Backend contract. It deliberately lives outside the
existing libkrun VM driver so the two runtimes can evolve independently.

The main OpenShell supervisor remains on the host. A private mode of the same
driver binary listens on the guest's virtio-vsock device and invokes the current
`openshell-supervisor-process` implementation only after the host advances the
boundary through `attach -> confirm -> start_agent`. The admitted policy crosses
that authenticated channel with the workload spec. The private protocol is not
a second public supervisor or agent API.

The prototype has no virtual NIC. This makes the network ceiling structurally
fail closed without TAP devices, nftables, `CAP_NET_ADMIN`, or `sudo`. Host
requirements are a Linux Firecracker binary and read/write access to `/dev/kvm`.

Current scope:

- boots an existing ext4 guest image with Firecracker;
- authenticates host-to-guest control over virtio-vsock;
- implements the RFC lifecycle and agent wait/signal operations;
- delegates guest process enforcement to the existing process supervisor leaf;
- serves the gateway compute-driver contract over a private Unix socket;
- provides an unprivileged KVM end-to-end smoke runner.

Exec, PTY, port forwarding, mediated guest egress, and per-connection binary
identity are intentionally deferred. The contract surfaces fail closed for
those operations. Code that might later become a shared VM helper is duplicated
here until a second consumer establishes a small, stable abstraction.

Run the smoke test with:

```shell
mise run e2e:firecracker
```

Start a plaintext local gateway backed by the driver with:

```shell
mise run gateway:firecracker
```

If the account is configured in the `kvm` group but the current process has
stale supplementary groups, the launcher re-enters that group with `sg`. It
does not use `sudo`.

On Linux toolchains where bindgen does not discover GCC's architecture-specific
headers, the task derives the include directory from
`gcc -print-file-name=include` and appends it to `BINDGEN_EXTRA_CLANG_ARGS`.

The gateway mode currently boots the configured rootfs fixture rather than
materializing the requested OCI image. Set `driver_config.command` to a string
array to override its default long-running shell workload.

See `e2e/firecracker/README.md` for fixture overrides.
Loading
Loading