feat(kubernetes): add proxy-pod supervisor topology - #2885
Open
russellb wants to merge 48 commits into
Open
Conversation
russellb
force-pushed
the
feat/kubernetes-proxy-pod-topology
branch
from
August 21, 2026 22:25
4308aa6 to
575a447
Compare
11 tasks
Add the Kubernetes proxy-pod topology with one supervisor Deployment and Service per sandbox, NetworkPolicy confinement, proxy-pod Helm/Skaffold configuration, topology documentation, and focused supervisor identity tests. Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
The proxy-pod agent egress NetworkPolicy hardcoded its DNS peers as kube-system/k8s-app=kube-dns and kube-system/k8s-app=coredns. That is an upstream Kubernetes convention, not a guarantee. On OpenShift, cluster DNS runs in the openshift-dns namespace with pods labeled dns.operator.openshift.io/daemonset-dns=default, and kube-system holds no DNS pods at all. The hardcoded selector matches nothing, so DNS egress falls through to the policy's implicit deny and the agent pod cannot resolve any name, including its own paired supervisor Service. The sandbox is inert. Add proxy_pod.dns_peers (Helm: supervisor.proxyPod.dnsPeers), a list of namespace/pod label selector pairs, defaulting to the previous upstream behavior so existing deployments are unaffected. Reject an empty peer list at startup, and render no DNS rule at all rather than an empty 'to' array when the list is empty: in NetworkPolicy semantics an empty 'to' matches every destination, so emitting one would silently open DNS-port egress cluster-wide. Signed-off-by: Russell Bryant <rbryant@redhat.com>
In proxy-pod topology the network supervisor runs in its own Deployment, so it does not stop when the agent pod does. A stopped sandbox kept its supervisor pod running indefinitely, consuming a pod slot, CPU, and memory for a sandbox the user believes is stopped. Scale the paired Deployment to zero on stop and back to one on start. The scale-down runs only after the workload has actually stopped so a graceful shutdown that needs egress still has it, and scaling failures are logged rather than failing the start/stop RPC. Extract the stop wait loop into wait_for_sandbox_stopped so the scale-down has a single place to hook, and grant the sandbox Role 'patch' on deployments. Signed-off-by: Russell Bryant <rbryant@redhat.com>
The Kubernetes driver assigns explicit non-root UIDs to sandbox and supervisor containers. OpenShift's restricted-v2 SCC uses runAsUser: MustRunAsRange and admits only UIDs inside the namespace's openshift.io/sa.scc.uid-range annotation, so it rejects both pods. The built-in nonroot-v2 SCC resolves this without a custom SCC: it is restricted-v2 with runAsUser: MustRunAsNonRoot and fsGroup: RunAsAny, while keeping requiredDropCapabilities ALL, allowPrivilegeEscalation false, no privileged containers, no host namespaces, and seccomp runtime/default. Its volume allowlist already covers every volume type proxy-pod topology uses. Add sandboxServiceAccount.openshift.nonrootSCC, default false so non-OpenShift installs never reference OpenShift-only APIs. When enabled it renders only a ClusterRole and ClusterRoleBinding granting 'use' on the existing nonroot-v2 SCC; no SecurityContextConstraints object is created. This makes proxy-pod the first OpenShell topology that runs on OpenShift under an unmodified, Red Hat-shipped SCC. Signed-off-by: Russell Bryant <rbryant@redhat.com>
A NetworkPolicy egress rule whose peer is a podSelector is evaluated against the destination pod after Service address translation, so the rule must carry the DNS pods' container port, not the Service port. Upstream CoreDNS listens on 53, so the two coincide. OpenShift's dns-default listens on 5353 and its Service maps 53 onto it, so a rule allowing port 53 never matches and the agent pod still cannot resolve anything. Verified on OpenShift 4.22 / OVN-Kubernetes: with the correct selectors but port 53, DNS failed both via the Service ClusterIP and via the DNS pod IP directly; with port 5353 it resolves. Add a per-peer 'port' field defaulting to 53, and emit one egress rule per peer rather than one shared rule, since a rule's port list applies to all of its 'to' entries and peers may differ. Signed-off-by: Russell Bryant <rbryant@redhat.com>
Update the RFC with what a live OpenShift 4.22 / OVN-Kubernetes deployment showed: the DNS peer port mismatch, the measured SCC split between the two pods, and two usability gaps that block adoption -- the user-supplied workload command is silently discarded, and sandboxes never leave Provisioning because nothing opens the supervisor session the Ready transition depends on. Replace the now-answered open question about OVN-Kubernetes service address translation with the questions those findings raise. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…ession The gateway forced SandboxPhase::Provisioning unless a ConnectSupervisor session was live. That session is opened only by openshell-supervisor-process and carries only relays -- SSH, exec, port forwarding, file transfer -- so proxy-pod topology, which has no in-sandbox process supervisor, could never reach Ready. Verified on OpenShift: both pods running and policy-enforced egress working end to end, while the sandbox reported Provisioning forever and every Ready-gated RPC, including stop and start, was unreachable. Add SupervisorSessionModel to DriverSandboxStatus. UNSPECIFIED preserves the existing contract, so drivers that never set it are unaffected. The Kubernetes driver reports NONE for proxy-pod and REQUIRED otherwise, and the gateway then derives readiness from the backend conditions alone. Ready must not become a lie in the process. The agent pod gains a wait-for-proxy init container that blocks on its paired supervisor's proxy port, so the pod is not Ready until egress actually works. This also closes a pre-existing ordering gap where the workload could start before the proxy existed and its early egress simply failed. Relay-backed RPCs now fail immediately with an explanation naming the topology instead of waiting out a session timeout that cannot succeed. Signed-off-by: Russell Bryant <rbryant@redhat.com>
proxy-pod runs the sandbox image directly, with no supervisor to launch a workload, so the container needs an entrypoint that stays running. OpenShell's own sandbox images use an interactive shell entrypoint, which reads EOF under kubelet and exits 0, leaving the pod in CrashLoopBackOff with empty logs. Add containers.agent.command and containers.agent.args to the Kubernetes driver_config passthrough, alongside the existing resources and volume_mounts. This needs no public API change: the initial command supplied to 'sandbox create' is delivered over the supervisor session, which this topology does not have. Reject the fields in combined and sidecar topology, where the driver replaces the container command with the supervisor binary and an override would be accepted and then silently dropped. Signed-off-by: Russell Bryant <rbryant@redhat.com>
Per-sandbox proxy-pod resources are named from the sandbox name, but the stop, start, and delete paths passed the Sandbox CR name. The two differ: a CR is named <workspace>--<sandbox>, so a sandbox named 'rdy' has CR 'default--rdy' and Deployment 'os-sup-rdy-<hash>'. The scale-down on stop therefore patched a Deployment that does not exist and silently did nothing, leaving the supervisor running for a stopped sandbox -- the exact problem the scaling was added to fix. Delete was affected too, but owner-reference garbage collection reclaimed the resources anyway and hid it. Read the sandbox name from the CR's sandbox-name label at both sites, and fall back to owner-reference GC with a warning if the label is missing. Caught by cluster testing; the unit tests passed throughout because they never exercised the CR-name-to-resource-name path. Signed-off-by: Russell Bryant <rbryant@redhat.com>
The earlier draft framed these as two independent gaps and said the driver silently discarded the workload command. That was wrong about the mechanism: the initial command from 'sandbox create' is delivered over the supervisor session after Ready, so it never ran because Ready never arrived. Rewrite both sections around what the code actually does -- Ready gated on a relay-carrying session that this topology cannot open -- and record the fixes and their cluster verification, including the CR-name versus sandbox-name bug that only on-cluster testing exposed. Signed-off-by: Russell Bryant <rbryant@redhat.com>
The feature list said SSH, exec, and file transfer were unavailable without saying why, which read as an implementation gap rather than a structural one. Record the mechanism: RelayOpen targets something 'inside the sandbox', the SSH server exists only in openshell-supervisor-process, and sessions need the workload's PID, mount, and network namespaces -- ssh.rs calls setns to enter the sandbox netns. The sidecar bridge to an abstract socket works only because both processes share a pod. Note that TCP relays are the exception and are recoverable for services bound to 0.0.0.0. Add the observability picture, measured on OpenShift. Network OCSF events, policy config events, and denial analysis all reach 'openshell logs' as usual, because log push is gated on sandbox ID and endpoint rather than topology. What is lost is workload stdout, which now reaches only the container log, and actor attribution on network events, which renders as -(0) because reading /proc across a pod boundary is impossible. Restructure the compatibility tables by concern and record the enforcement mechanism, pods per sandbox, and OpenShift SCC per topology. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…rror Creating a sandbox in a topology with no in-sandbox supervisor succeeded, then failed at the interactive-session step with a bare gRPC error. The sandbox was running and its network policy enforced, but the output read as a failed create. Detect the gateway's rejection and print what actually happened: the sandbox is running, sessions are unavailable for this topology, egress is unaffected, and which topologies to use when interactive access is required. When a command was passed to 'sandbox create', say plainly that it did not run and point at the containers.agent.command entrypoint override instead. The command still exits non-zero. A command that did not run must not report success, and callers should not have to parse output to find that out. Detection keys on a stable marker constant shared through openshell-core rather than on prose, so rewording the message cannot silently break it, and it searches the whole error chain because the marker arrives wrapped in a transport error. Signed-off-by: Russell Bryant <rbryant@redhat.com>
The previous commit explained the topology after a relay RPC was rejected, but 'sandbox create' still spawned ssh first, so the failure surfaced through the subprocess as 'ssh exited with status 255' and the explanatory message was buried in wrapped stderr. Publish a SupervisorSession=False/NotApplicable condition in the public sandbox status when the driver reports SupervisorSessionModel::None, and have the CLI check it before attempting a session. When set, the CLI skips the connect/exec path entirely and prints the explanation directly: the sandbox is running, sessions are unavailable for this topology, egress is unaffected, and how to set a workload entrypoint when a command was supplied. The error-marker detection from the prior commit stays as the backstop for relay RPCs issued directly against an existing sandbox, where there is no create-time status to pre-check. Signed-off-by: Russell Bryant <rbryant@redhat.com>
Add the workload-to-supervisor kernel-isolation property to the topology comparison: because proxy-pod places the workload and supervisor in separate pods, a VM-based RuntimeClass like Kata gives them separate VMs and kernels, so a workload kernel compromise does not by itself reach the supervisor's gateway credentials. This is unique to proxy-pod; the in-pod topologies share one Kata VM between workload and supervisor. Clarify that lost workload stdout/stderr is specifically the agent container's log, reachable only via 'kubectl logs <agent-pod>', not 'openshell logs'. Signed-off-by: Russell Bryant <rbryant@redhat.com>
Main renamed the canonical-command transport from OPENSHELL_SANDBOX_COMMAND to the versioned OPENSHELL_MAIN_PROCESS_SPEC (NVIDIA#2726), which the supervisor decodes and launches. proxy-pod runs the sandbox image directly with no supervisor, so that env var is not only useless in the workload container but leaks the intended command into it. Strip MAIN_PROCESS_SPEC alongside the other supervisor-oriented variables, replacing the now-removed SANDBOX_COMMAND entry. Rebase adaptation: proxy-pod's workload command continues to flow through the containers.agent.command driver_config, since the canonical main process requires an in-sandbox supervisor this topology does not run. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…ps the command A non-interactive persistent create takes main's implicit-detach path and returns before the sessionless check ran, so 'openshell sandbox create -- cmd' against a proxy-pod sandbox silently created a sandbox where the command never runs -- exactly the broken-looking outcome the sessionless messaging exists to prevent. Check for a command-bearing sessionless topology before the detach return and explain that the command will not run, pointing at containers.agent.command. The interactive no-command case still reports after detach. Both paths share a new abort_sessionless_create helper so ephemeral cleanup and messaging stay identical. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…d placement Addresses several proxy-pod review findings in the Kubernetes driver: - Delete the Sandbox CR (tearing down the workload) before removing the companion resources, so the agent egress NetworkPolicy fence is never dropped while the workload can still egress, and a failed CR delete leaves the fence in place. - Derive companion resource names from the Sandbox CR name, which is unique in every workspace mode, instead of the bare sandbox name. In shared mode workspace-a/dev and workspace-b/dev share a sandbox name, so the previous scheme collided and a rollback could dismantle another sandbox's isolation. - Create companions in, and point the workload's proxy URL at, the sandbox's resolved target namespace rather than the static configured namespace, so proxy-pod works in managed and operator workspace modes. Add the proxy-pod resources to the cluster-scoped Role for those modes. - Remove the raw gateway-forward tunnel (supervisor :18080 to the gateway, reachable by the agent). Nothing on the agent consumed it, and with unauthenticated gateway access it was a policy-bypassing path to the admin API. The supervisor still connects to the gateway directly for its own policy, inference, and log traffic. - Return an error from supervisor Deployment scaling and propagate it from start_sandbox, so a transient scale-up failure surfaces instead of wedging the sandbox in Starting with the supervisor at zero replicas. Scale-down on stop stays best-effort. - Give the supervisor Deployment the workload's nodeSelector, tolerations, and priorityClassName, so required same-node affinity cannot pin the workload to a node its own placement excludes. Signed-off-by: Russell Bryant <rbryant@redhat.com>
set_sessionless recorded proxy-pod sandboxes in the supervisor session registry, but forget_sessionless was never called, so the set grew without bound as ephemeral proxy-pod sandboxes were created and deleted. Clear the marker in cleanup_sandbox_state, which runs on permanent removal. It is deliberately not cleared in the stopped-session cleanup: the sessionless property is a topology fact that must survive stop/start. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…teps The sessionless check ran after the structured-output early return and after the upload, forward, and editor steps. So sandbox create --output json -- <cmd> exited zero while proxy-pod silently discarded the command, and upload, forward, or editor failures on --no-keep bypassed cleanup and leaked the ephemeral sandbox. Detect the sessionless topology at the top of the Ready arm. When a session-requiring operation was requested (a command, upload, forward, or editor), abort immediately -- cleaning up an ephemeral sandbox and reporting a non-zero exit -- before any of those steps or a structured-success print. A bare create with no such operation still succeeds (the network-only sandbox is created), emitting structured output when requested and detaching otherwise. Signed-off-by: Russell Bryant <rbryant@redhat.com>
The e2e:kubernetes:proxy-pod task ran the generic Kubernetes suite, whose smoke test execs a command and reads its output -- capabilities proxy-pod lacks, so the suite could not pass. Add tests/proxy_pod.rs (feature e2e-kubernetes-proxy-pod) covering the topology's actual contract: a workload whose entrypoint is set through containers.agent.command reaches Ready, and relay-backed operations (exec) are rejected with a topology-specific error. Scope the task to this suite instead of the incompatible generic one. The NetworkPolicy egress boundary is asserted at the unit level in the driver and validated manually on a policy-enforcing cluster; a self-probing egress e2e needs a workload image that tests its own egress and reports through 'openshell logs', tracked as follow-up. Signed-off-by: Russell Bryant <rbryant@redhat.com>
Update the topology diagram, NetworkPolicy contract, and validation table now that the supervisor no longer forwards a raw gateway tunnel, and note in the credential-isolation section that the workload has no network path to the gateway at all. Signed-off-by: Russell Bryant <rbryant@redhat.com>
… template The companion resources are named from the Sandbox CR name, but the workload pod template still derived the CA secret mount and HTTP_PROXY Service name from the bare sandbox name, and the create-rollback path cleaned up bare-name resources in the static namespace. In shared mode the workload then mounted a CA secret that did not exist and never became Ready. Thread the CR name through SandboxPodParams and use it for the pod template's companion references and the rollback cleanup, matching create_proxy_pod_resources. Caught by re-testing a shared-mode create on the cluster. Signed-off-by: Russell Bryant <rbryant@redhat.com>
Address code review feedback on the proxy-pod topology: - Clean up companions only on a confirmed CR delete, never on a 409 (replacement) or 404, so a concurrent replacement keeps its Deployment, Secret, Service, and egress NetworkPolicy. - On failed companion creation, delete the CR by its returned name with a UID precondition before removing companions, so shared mode no longer targets the wrong CR and leaves the workload unfenced. - Persist the creation-time supervisor topology on each Sandbox CR and derive status and start/stop/delete behavior from it, so a later gateway topology change does not reinterpret existing sandboxes. - Mirror the workload's public platform_config placement (runtime class, node selector, tolerations) onto the supervisor Deployment so same-node affinity stays schedulable and runtimes match. - Grant proxy-pod ClusterRole RBAC in operator mode, not just managed. - Restrict the OpenShift nonroot-v2 SCC option to shared workspace mode and document the constraint; fail the Helm render otherwise. - Correct the debug skill to reference only supervisor ingress port 3128 and note workspace-mode RBAC scoping. Signed-off-by: Russell Bryant <rbryant@redhat.com>
russellb
force-pushed
the
feat/kubernetes-proxy-pod-topology
branch
from
August 24, 2026 13:00
3f1f2f2 to
492ff77
Compare
Address round-two review feedback: - Scope the proxy-pod ClusterRole to least privilege: grant Secrets (and Services and NetworkPolicies) create/delete only, matching the namespaced Role. The gateway never reads Secrets, so it no longer holds cluster-wide Secret read access. - Stop eagerly deleting companion resources on sandbox delete and on create-failure rollback. Every companion is owner-referenced to the Sandbox CR, so garbage collection removes them as part of the CR teardown; deleting the egress NetworkPolicy eagerly raced the workload pod's termination grace period and could reopen unrestricted egress. - Resolve the supervisor Service with a search-domain-relative name instead of a hardcoded .svc.cluster.local, so clusters with a custom cluster domain can resolve it. - Note in the openshell-cli skill that proxy-pod topology is sessionless and rejects trailing commands, uploads, connect, and exec. - Remove a duplicated proxy_pod.proxy_uid row from the compute-drivers doc. Signed-off-by: Russell Bryant <rbryant@redhat.com>
A gateway crash between the Sandbox CR create and its companion creates left a persisted CR with a partial topology that ordinary reconciliation never repaired, because the CR already existed. - Apply companions with create-if-absent semantics (AlreadyExists treated as success), so provisioning is idempotent and additive: an existing CA Secret keeps its key material and an existing supervisor Deployment keeps its replica count. - Reconcile companions for every existing proxy-pod Sandbox CR on each watch_sandboxes call (gateway start and watch re-establishment), rebuilding the render inputs — placement and log level — from the CR's own agent pod so a repaired supervisor lands where the workload can pair with it. - Share the SandboxPodParams builder between the create and reconcile paths so both render identical companions. Validated on OpenShift/OVN-Kubernetes: deleting a supervisor Deployment and restarting the gateway recreates it (checked=1 failed=0), and deleting a sandbox garbage-collects all companions via owner references. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…ned topology In combined topology the network supervisor shares the workload's container and inherits the workload image's baked-in environment. Honoring OPENSHELL_PROXY_BIND_ADDR there let an untrusted image publish the credential-bearing policy proxy on the pod network (e.g. 0.0.0.0:3128), turning the sandbox into a confused deputy; OPENSHELL_PROXY_CA_CERT_PATH/KEY_PATH similarly let an image substitute an attacker-controlled CA. Gate both on a trusted launch context: only a standalone network supervisor (proxy-pod/sidecar; process supervision not co-located) may take its bind address and CA files from the environment, which the driver sets in a separate container built from the trusted supervisor image. A combined supervisor now binds to the namespace-scoped veth IP and always generates an ephemeral CA, ignoring image-supplied values. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…rd upstream proxy, track supervisor readiness Address round-three review warnings: - Key companion resource names on the immutable sandbox UUID with a 64-bit suffix instead of a 32-bit hash of the truncatable CR name, which had a deterministic collision path. Distinct sandbox instances now never share a companion name, closing the stale-object reuse window. - On an AlreadyExists (409) conflict, fetch the object and confirm it is owned by the same Sandbox CR before treating the create as idempotent; fail closed when a different instance owns it, so a new sandbox never adopts a stale companion. - Forward the operator's corporate upstream proxy from the proxy-pod supervisor (URL, no_proxy, CONNECT mode), matching sidecar topology, so egress is not silently dropped or routed around the required monitoring path. Credentials are excluded because the supervisor pod does not mount the auth Secret. - Fold supervisor Deployment availability into sandbox status: a proxy-pod sandbox whose supervisor has no available replica reports NotReady (SupervisorUnavailable) rather than staying Ready with a dead egress path. Also updates the debug skill and the proxy-pod RFC risks to reflect these mitigations. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…conciliation Address round-four review (RBAC and reconciliation correctness): - Grant no `delete` on proxy-pod companion Deployments, Services, Secrets, or NetworkPolicies. The gateway never deletes companions — garbage collection removes them with the Sandbox CR — so these verbs were unused and let a compromised gateway delete arbitrary cluster resources (Critical 1). - Grant `get` on the non-secret companions (Services, NetworkPolicies; already present for Deployments) so crash-recovery reconciliation can verify an existing companion's owner before adopting it. Previously reconciliation's 409-verification GET hit a 403 and stopped, so partial companion sets were never repaired (Warning 1). - Keep Secrets at `create` only: the gateway never reads Secret contents. The CA Secret skips ownership verification on 409 (its UUID-keyed name already implies it is this sandbox's own), and ownership verification treats a 403 as "accept" rather than wedging reconciliation. - Scope companion reconciliation to the gateway's own sandboxes and drive it from each CR's persisted creation-time topology instead of the gateway's current config, so one gateway never repairs another's resources and a gateway reconfigured to `combined` still reconciles pre-existing proxy-pod sandboxes (Warning 2). - Correct the compute-drivers reference: proxy-pod runs no in-pod supervisor, is sessionless, and its workload logs are not in `openshell logs`. Signed-off-by: Russell Bryant <rbryant@redhat.com>
… exit Owner-reference garbage collection does not order sibling deletion, so a fence owned by the Sandbox CR was removed concurrently with the workload pod; a pod that ignores SIGTERM could regain direct egress during its termination grace period (CWE-693). Make the agent egress NetworkPolicy gateway-managed instead of GC-owned: - Create it with no ownerReference so garbage collection never removes it. - On delete, delete the Sandbox CR, wait for the workload pod to disappear, then delete the fence explicitly. If the pod cannot be confirmed gone, leave the fence for reconciliation rather than dropping it on a guess. - Reap orphaned fences in reconciliation (an os-eg-* policy whose Sandbox CR no longer exists), covering a gateway crash between CR deletion and fence teardown. On create-failure rollback, delete the ownerless fence explicitly. - Grant delete/list on networkpolicies for this gateway-managed lifecycle; the other companions remain GC-owned with no delete grant. Updates the proxy-pod RFC and debug skill to describe the split teardown. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…en reconciliation Address round-five review of the proxy-pod egress fence and reconciliation: - Never drop the egress fence until the workload pod is confirmed gone. The create-failure rollback and the reconciliation orphan reaper both now check for the agent pod (by immutable sandbox-id + agent-role labels) and retain the fence when its absence cannot be confirmed, closing the window where a SIGTERM-ignoring workload regained direct egress (findings 1, 2). - Validate an existing ownerless egress fence on AlreadyExists instead of accepting it blindly: fetch it and require its spec and sandbox-id to match the intended fence, failing closed on a stale or altered policy (finding 4). - Never classify an un-annotated (pre-branch) CR as proxy-pod. Such CRs predate the topology annotation that every proxy-pod sandbox carries, so the fallback collapses to combined rather than the gateway's current topology, avoiding misclassifying existing combined/sidecar sandboxes on upgrade (finding 3). - Reconcile the supervisor Deployment replica count from the CR's operating state (Running -> 1, Suspended -> 0) so a crash between the operating-state patch and the scale is repaired, rather than recreating a missing Deployment with one replica unconditionally (finding 6). - Fold supervisor Deployment availability into the watch paths too, so a CR event never republishes a sandbox as Ready while its supervisor is down (finding 7). Grants pods `list` (proxy-pod only) for the label-based pod-absence checks. Signed-off-by: Russell Bryant <rbryant@redhat.com>
The exec and port-forwarding handlers remapped every relay error to Unavailable, which the CLI treats as transient and retries. A sessionless (proxy-pod) sandbox returns FailedPrecondition, a terminal condition that can never succeed, so standalone port forwarding stayed open retrying forever. Preserve the original status code (the CLI already treats FailedPrecondition as fatal). Signed-off-by: Russell Bryant <rbryant@redhat.com>
…contract The Helm unit tests still asserted delete permissions and a combined Service/Secret rule that the templates intentionally dropped, so `mise run helm:test` was red. Update the assertions to the least-privilege contract (no companion delete, split Service/Secret, Secret create-only, NetworkPolicy create/delete/get/list for the gateway-managed fence) and correct the obsolete permission contract in the gateway architecture doc. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…n, and scoped pod checks Address round-six review of the proxy-pod fence and RBAC: - Tear the egress fence down on create-failure rollback only after the Sandbox CR deletion is confirmed (success or 404) AND the workload pod is gone. A CR delete that never reached Kubernetes previously let teardown remove the fence while the surviving CR could still create an unfenced workload (finding 1). - Never treat a vanished fence as provisioned: when create returns 409 but the verifying GET returns 404, re-create it rather than returning success, so the workload is never left at default-allow (finding 2). - Add supervisor.proxyPod.retainCompanionRbac to keep companion/fence RBAC while migrating a gateway away from proxy-pod with proxy-pod sandboxes still present, instead of stripping the permissions their lifecycle needs (finding 3). - Confirm workload-pod absence with a name-scoped get instead of a cluster-wide pods list: the fence records the guarded pod's name (== its CR name) so delete/reap address it exactly. Drops the pods:list grant that RBAC could not constrain to a selector (finding 4). Signed-off-by: Russell Bryant <rbryant@redhat.com>
… status Under HA, only the reconciler lease holder populated the in-memory sessionless set, so relay-backed RPCs (exec, interactive exec, port forwarding) routed to another gateway replica waited out the 15s session timeout and returned a transient Unavailable that the CLI retries — leaving port forwards open against a sandbox that can never serve them. The exec/interactive/forward handlers now reject with FailedPrecondition from the durable SupervisorSession=NotApplicable condition on the stored status, which every replica can read. Signed-off-by: Russell Bryant <rbryant@redhat.com>
- Show the required openshift-dns:5353 DNS peer and nonroot-v2 SCC grant in the proxy-pod topology example, plus the retainCompanionRbac migration note. - Point the debug skill at the separate supervisor Deployment for proxy-pod supervisor logs; the sandbox pod has only the workload agent container. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…ch supervisor Deployments Companion reconciliation previously ran only when the sandbox watch was established, so a stop-time supervisor scale-down that failed transiently (or an egress fence orphaned by a gateway crash) persisted until the watch re-established. Add a periodic reconcile bound to the watch's lifetime that re-runs companion reconciliation every 30s, correcting supervisor replica drift and reaping orphaned fences without waiting for the watch to drop. Supervisor Deployment availability was also not observed: readiness only refreshed on get/list queries and the reconcile sweep, so a supervisor that went unavailable after startup could leave the sandbox reporting Ready for up to a full sweep. Watch supervisor Deployments (gateway- and role-scoped) and push a refreshed sandbox status within seconds of an availability change; direct queries and the periodic reconcile remain as a backstop. Grant list and watch on apps/deployments in the proxy-pod Role and ClusterRole to back the Deployment watch, and update the design and operator docs accordingly. Signed-off-by: Russell Bryant <rbryant@redhat.com>
Wire the capability-scoped proxy_pod suite into branch E2E as kubernetes-proxy-pod-e2e and gate the core-e2e-result job on it. CI's kind cluster uses a non-enforcing CNI, so this exercises the proxy-pod control-plane contract (companion creation, readiness, sessionless relay rejection); the CNI-enforced egress isolation test still needs a policy-enforcing CNI and remains tracked as follow-up. Signed-off-by: Russell Bryant <rbryant@redhat.com>
The helper is declared in a private module, so pub(crate) is redundant and trips clippy::redundant_pub_crate under -D warnings. Match the neighboring SUPERVISOR_SESSION_CONDITION visibility. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…panionRbac Regenerate the chart README so helm-docs check passes; the retainCompanionRbac value row was missing. Signed-off-by: Russell Bryant <rbryant@redhat.com>
… scoping Close a fence-reaping race, stop supervisor readiness failing open, keep management alive during a retainCompanionRbac migration, and drop cluster-wide Deployment enumeration. - Reap race: the periodic reconcile snapshots live CR ids before listing egress policies, so a sandbox created in that window (CR then fence, in that order) looked orphaned and its fresh fence could be deleted before its workload pod existed, leaving the workload with default-allow egress. Re-confirm the Sandbox CR is absent immediately before deleting; retain on "exists" or "unknown". - Fail-open readiness: supervisor availability is now tri-state (Available/Unavailable/Unknown). The Deployment watch derives availability from the event object itself instead of re-fetching (a GET that could time out and republish a dead-egress sandbox as Ready); a definite Unavailable is required to downgrade readiness, and Unknown leaves it unchanged. - Migration: periodic reconciliation and (shared mode) the supervisor Deployment watch now run whenever the gateway manages proxy-pod sandboxes - either its configured topology is proxy-pod, or a retainCompanionRbac migration left proxy-pod sandboxes it still owns - determined from the startup reconcile. The Deployment watch degrades to reconcile-only on error instead of tearing down the sandbox watch. - Least privilege: the supervisor Deployment watch runs only in shared (single-namespace) mode via the namespaced Role. Managed/operator modes fold readiness in through get/list and the periodic reconcile, so list/watch on apps/deployments is removed from the cluster-wide ClusterRole. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…cable reason Interactive (explicit-TTY) exec opened a relay without the durable sessionless check, so on a follower replica it waited out the 15s relay-open timeout and returned retryable Unavailable instead of the terminal FailedPrecondition the unary exec and TCP forwarding paths already return. Reject it up front like the others. Also require reason=NotApplicable (not merely status=False) when reading the durable SupervisorSession condition, so a future driver that reports SupervisorSession=False for a transient disconnect is not given a terminal relay rejection. The producer already sets that reason; a shared constant now ties the two together. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…ance - openshift.mdx: privileged SCC is the default (combined) topology's requirement; sidecar and proxy-pod run under the built-in nonroot-v2 SCC. - gateway-config.mdx: document proxy_pod.dns_peers with the OpenShift example. - debug-openshell-cluster skill, gateway.md, and the RFC: reflect that the supervisor Deployment watch (and its list/watch on apps/deployments) is shared-mode-only, and that retainCompanionRbac keeps proxy-pod sandboxes managed through a topology migration. Signed-off-by: Russell Bryant <rbryant@redhat.com>
russellb
marked this pull request as ready for review
August 25, 2026 16:24
russellb
requested review from
a team,
derekwaynecarr,
mrunalp and
sjenning
as code owners
August 25, 2026 16:24
Contributor
Author
|
I'm still running local review, but I think this is ready to look at and try out |
…eadiness closed Two correctness fixes in the proxy-pod driver: - Fence teardown treated an accepted Sandbox CR DELETE as proof the CR was gone. Kubernetes may only have set deletionTimestamp; a finalizer or in-flight controller reconciliation can still recreate the workload after the momentary pod-absence check, and the fence would already be removed — default-allow egress. Before deleting the fence, re-confirm the UID-addressed CR is actually absent (or replaced by a different-UID successor); retain the fence otherwise so reconciliation reaps it once the CR is truly gone. - Unknown supervisor availability failed open. A failed Deployment GET left the CR's own Ready=True intact, so a watch event or periodic list could overwrite a prior DependenciesNotReady with Ready even though the separate supervisor may be down. Fail closed: keep Ready only when the supervisor is confirmed Available; both Unavailable and Unknown downgrade to Provisioning. Signed-off-by: Russell Bryant <rbryant@redhat.com>
…e discovery Whether periodic companion reconciliation and the shared-mode supervisor Deployment readiness watch stay scheduled was inferred from a runtime sandbox list at watch establishment. One transient discovery failure returned "manages none" and froze that decision for the entire watch session, disabling migration repair for otherwise-healthy proxy-pod sandboxes after the configured topology was switched away from proxy-pod. Decide it from configuration instead: add proxy_pod.retain_companion_management (rendered by Helm from supervisor.proxyPod.retainCompanionRbac) and schedule upkeep when the configured topology is proxy-pod OR that flag is set. The watch-establishment reconcile still runs for its repair side effects; its result no longer gates upkeep, so a flaky list can never disable it. Signed-off-by: Russell Bryant <rbryant@redhat.com>
Only proxy-pod runs sandbox pods under the built-in nonroot-v2 SCC. The sidecar and cni-sidecar topologies need a custom SCC — their UID-0 network init container and default root sidecar require added capabilities — so nonroot-v2 would reject them. The prior text wrongly lumped sidecar in with proxy-pod. Signed-off-by: Russell Bryant <rbryant@redhat.com>
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the Kubernetes
proxy-podsupervisor topology: network enforcement andgateway forwarding move out of the sandbox pod entirely and into a paired,
per-sandbox supervisor
Deployment. The sandbox pod runs the agent imagedirectly — no supervisor binary, no gateway credentials, no privileged init
container, no shared process namespace. Egress is fenced by two per-sandbox
Kubernetes
NetworkPolicyobjects rather than by pod-local nftables rules.The result is the least-privileged sandbox pod any OpenShell topology produces:
runAsNonRoot, all Linux capabilities dropped, no added privilege at any layer.That is what makes it the first topology to run on OpenShift under a stock,
Red Hat-shipped SCC (
nonroot-v2) with no bespoke security grant.This continues @TaylorMutch's original proxy-pod work in #2077 (and the
earlier #2016), rebased onto current
mainwith the correctness,OpenShift-enablement, readiness, and workload-command work needed to make the
topology usable. It is the sibling of the
cni-sidecartopology in #2606.Topology tradeoffs (combined / sidecar / cni-sidecar / proxy-pod)
The topologies trade privilege against capability. The dividing line for
proxy-podis consistent: everything enforceable or observable at the networkboundary is retained, and everything requiring visibility inside the
workload's namespaces is given up.
In every topology the network supervisor (the OpenShell proxy) is the egress
policy engine — it evaluates L4/L7 network policy, terminates TLS for L7
inspection, injects credentials, and makes the policy-approved upstream
connections. The topologies differ in how the workload is confined to that
proxy so it cannot route around it, and — for
proxy-pod— in where thesupervisor runs. In
proxy-podthe supervisor is a separate per-sandbox podand the confinement is a Kubernetes
NetworkPolicypair: an agent-egresspolicy that lets the workload pod reach only the supervisor (plus cluster DNS),
and a supervisor-ingress policy that lets only the paired workload reach the
supervisor's proxy port. The in-pod topologies enforce that confinement with
node-local nftables/CNI rules, so it holds regardless of CNI;
proxy-podreliesentirely on CNI
NetworkPolicyenforcement (see the RequiresNetworkPolicyenforcement row), which is why a non-enforcing CNI is its highest-severity risk.
combined(default)sidecarcni-sidecarproxy-podNetworkPolicyexec/ upload / sync0.0.0.0binds)openshell logsopenshell logsopenshell logsagentcontainer log (kubectl logs <agent-pod>), notopenshell logs-(0))NetworkPolicyenforcementprivilegedprivilegedCNInonroot-v2When to use which.
combinedstays the default and the only topology withthe full supervisor contract.
sidecar/cni-sidecarkeep that contract(including SSH, exec, and filesystem policy) while lowering pod privilege, at the
cost of a per-pod privileged init container or a node-level CNI DaemonSet.
proxy-podis for clusters that will not admit in-pod privilege at any level andfor workloads that need policy-enforced egress but never an interactive session —
batch jobs and autonomous agents that ship their own long-running entrypoint.
Operators who want the interactive workflow and low pod privilege on
OpenShift should use
cni-sidecar(#2606), notproxy-pod. The two arecomplementary, not competing.
proxy-podalso raises the isolation ceiling under a VM-basedRuntimeClass:because the workload and supervisor are in separate pods, Kata Containers
places them in separate VMs with separate kernels. A kernel compromise in the
workload VM does not by itself reach the supervisor or its gateway credentials.
In the in-pod topologies the two share one Kata VM, so the boundary between them
is a namespace boundary, not a hypervisor one. This is unique to
proxy-pod.Why SSH/exec are structurally impossible here, not merely unimplemented: every
relay targets something inside the sandbox, the SSH server lives only in the
process supervisor, and sessions need the workload's PID/mount/network
namespaces (
ssh.rscallssetnsto enter the sandbox netns). A supervisor ina separate pod holds none of those. Full analysis in the RFC.
RFC
rfc/proxy-pod-topology-DRAFT.md(included in this PR, unnumbered pending amaintainer-assigned number). It covers:
driving case.
proxy CA Secret, two NetworkPolicies), the privilege and credential-isolation
model, and the NetworkPolicy egress/ingress contract.
configurable (OpenShift runs DNS in
openshift-dnson container port 5353,not
kube-system/53), and why the built-innonroot-v2SCC suffices.supervisor to launch it — the two adoption blockers found during cluster
validation, with the fixes.
structural feature and telemetry losses, measured on-cluster.
Open questions.
The RFC records results measured against a live OpenShift 4.22 / OVN-Kubernetes
cluster, including the SCC split (the agent pod admits under stock
restricted-v2; only the supervisor needsnonroot-v2).Related Issue
Continues @TaylorMutch's proxy-pod PR #2077, which references #1827, #981,
#899, and #1305. Maintainers assign the RFC number from the originating issue
before it leaves draft.
Changes
Topology (builds on @TaylorMutch's #2077):
proxy-podsupervisor topology: per-sandbox supervisorDeployment,headless
Service, generated proxy CASecret, and agent-egress /supervisor-ingress
NetworkPolicypair. The Deployment, Service, Secret, andsupervisor-ingress policy are owner-referenced to the Sandbox CR and
garbage-collected with it. The agent-egress fence is deliberately not
owner-referenced: Kubernetes GC does not order sibling deletion, so a GC-owned
fence could be removed alongside the workload pod and let a
SIGTERM-ignoringworkload regain direct egress during its grace period. The gateway instead
manages the fence directly — deleting it only after the workload pod is gone,
and reaping any fence orphaned by a gateway crash.
dropped; inject only proxy and CA-trust environment.
readability), so they stay stable and unique across sandbox-name reuse.
proxy_pod.proxy_uid/proxy_pod.affinityconfiguration and Helmvalues.
OpenShift enablement (new):
proxy_pod.dns_peers, Helmsupervisor.proxyPod.dnsPeers) with per-peer namespace/pod selectors andport, defaulting to the upstream
kube-systemconventions. An empty listis rejected, and an empty peer renders no rule rather than an allow-all rule.
nonroot-v2SCC grant (sandboxServiceAccount.openshift.nonrootSCC,default off) — a ClusterRole/Binding only, no custom SCC object.
Readiness and workload command (new):
SupervisorSessionModelon the driverDriverSandboxStatuscontract(
UNSPECIFIEDpreserves existing behavior). The gateway derives readinessfrom backend conditions when a topology reports it runs no in-sandbox
supervisor.
wait-for-proxyagent-pod init container (newwait-for-tcpsupervisorsubcommand) so pod readiness transitively means egress works.
Deployment's live availability into sandbox readiness: aproxy-pod sandbox whose supervisor has no available replica reports
Provisioning(not a staleReady) and recovers when the supervisor does. Inshared mode a supervisor
Deploymentwatch pushes that status within seconds;every mode's get/list and the periodic reconcile fold the same check as a
backstop. Availability is tri-state, so a transient Deployment-GET error never
fails open to
Ready.containers.agent.command/argsvia the Kubernetesdriver_configpassthrough (no public API change), rejected in
combined/sidecarwhere thesupervisor is the entrypoint.
Lifecycle & HA hardening (from review):
establishment, corrects supervisor replica drift (e.g. a transiently-failed
stop-time scale-down leaving a stopped sandbox's supervisor running) and reaps
egress fences orphaned by a gateway crash — re-confirming the Sandbox CR is
gone immediately before deleting a fence, so a newly created sandbox never
loses its fresh fence to the reaper.
SupervisorSession=NotApplicableis published as a durable status condition soevery gateway replica (not just the reconciler lease holder) rejects
relay-backed RPCs — unary
exec, interactiveexec, and TCP forwarding —with an immediate, terminal error instead of a 15s relay timeout.
supervisor.proxyPod.retainCompanionRbackeeps the companion RBAC,periodic reconcile, and readiness watch working for existing proxy-pod
sandboxes after
supervisor.topologyis switched away from proxy-pod; thedriver manages them by their persisted creation-time topology.
deleteonowner-referenced companions. The supervisor
Deploymentreadiness watch (andits
list/watchonapps/deployments) is granted only through thenamespaced Role in shared mode; managed/operator modes fold readiness via
get/list to avoid cluster-wide Deployment enumeration.
Correctness fixes (found during rebase and cluster testing):
proxy-pod(would landin the workload pod).
Deploymentto zero onstopand back onstart(anearlier revision that keyed names on the CR name silently scaled a nonexistent
Deployment —
default--rdyvsrdy— now fixed by the UUID-keyed namingabove).
SupervisorSessionstatus condition, with an error-marker backstop) and print a clear
explanation instead of
ssh exited with status 255.Docs: RFC,
docs/kubernetes/topology.mdx,docs/kubernetes/openshift.mdx,docs/reference/sandbox-compute-drivers.mdx,docs/reference/gateway-config.mdx,Helm README, and the
debug-openshell-cluster/helm-dev-environmentskills.Testing
cargo test -p openshell-driver-kubernetes(251) and-p openshell-server --lib(1427).mise run helm:test— 120 pass (proxy-pod DNS-peer, SCC-grant, RBAC, andretainCompanionRbaccases added).mise run pre-commit— clean.proxy_pode2e suite runs in branch CI(
kubernetes-proxy-pod-e2e, on a kind cluster) covering the control-planecontract (companion creation, readiness, sessionless relay rejection). The
CNI-enforced egress-isolation assertions remain a tracked follow-up, since
kind's default CNI does not enforce
NetworkPolicy.Ready; unproxied egress denied and proxied egress policy-evaluated(allow → 200, deny → 403 at CONNECT) with the generated CA trusted; DNS
resolves via
openshift-dns:5353;stop/startscale the supervisordown/up; the stock base image runs via
containers.agent.command; both podsadmit (agent under
restricted-v2, supervisor undernonroot-v2). Reviewhardening re-validated on-cluster: scaling the supervisor to zero flips the
sandbox to
Provisioningwithin ~1s and it recovers toReady; the periodicreconcile restores a stopped sandbox's errantly-running supervisor and a
running sandbox's dropped replica; a genuinely orphaned egress fence is reaped
while a live sandbox's fence is retained; unary and interactive (
--tty)execare both rejected immediately with the topology error; theowner-referenced companions are GC'd on delete and the gateway-managed egress
fence is torn down after the workload pod exits.
Checklist