diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index 80d5057b17..04956d3991 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -392,7 +392,7 @@ kubectl -n openshell get pod -l app.kubernetes.io/name=helm-chart -o jsonpath="{ ``` Sandbox pods using provider token grants should have an -`openshell.io/sandbox-id` annotation, an `openshell.ai/managed-by=openshell` +`openshell.ai/sandbox-id` annotation, an `openshell.ai/managed-by=openshell` label, supervisor env vars `OPENSHELL_K8S_SA_TOKEN_FILE` and `OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET`, plus both the projected `openshell-sa-token` volume and the `spiffe-workload-api` CSI volume. @@ -467,7 +467,10 @@ Then inspect sandbox resources in that namespace. Check the configured sandbox service account when TokenReview bootstrap or sandbox registration fails. Helm creates a dedicated sandbox service account by default and writes it to `[openshell.drivers.kubernetes].service_account_name`; -the gateway rejects projected tokens from other service accounts. +the selected Kubernetes compute driver rejects projected tokens from other +service accounts. For an external driver, inspect its logs and confirm it +advertises `supports_sandbox_authentication`; the gateway delegates the opaque +credential over the driver socket and never interprets Kubernetes settings. ```bash helm -n openshell get values openshell | grep -A3 sandboxServiceAccount diff --git a/.agents/skills/helm-dev-environment/SKILL.md b/.agents/skills/helm-dev-environment/SKILL.md index ebd9a01595..c48d94b5d3 100644 --- a/.agents/skills/helm-dev-environment/SKILL.md +++ b/.agents/skills/helm-dev-environment/SKILL.md @@ -256,7 +256,9 @@ annotations to `spiffe://openshell.local/openshell/sandbox/`. OpenShell mounts the SPIFFE CSI Workload API socket at `/spiffe-workload-api/spire-agent.sock` into sandbox pods for provider token grants. Supervisor-to-gateway authentication remains on the Kubernetes -ServiceAccount bootstrap and gateway-minted sandbox JWT path. +ServiceAccount bootstrap and gateway-minted sandbox JWT path; the selected +Kubernetes compute driver validates the projected token before the gateway +mints its JWT. --- diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index 2a36073486..b90778587d 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -169,6 +169,15 @@ The driver reports this behavior through in-process and external drivers. Older drivers omit the field and retain the conservative operator-managed behavior. +Drivers that can verify a platform-native sandbox credential advertise +`GetCapabilities.supports_sandbox_authentication`. On the path-scoped +`IssueSandboxToken` exchange, the gateway forwards the opaque bearer credential +to that selected driver through `AuthenticateSandbox`. The driver returns only +the authenticated sandbox ID. The gateway then verifies that its durable +sandbox record exists and mints the gateway JWT. The driver socket is therefore +a sandbox-identity trust boundary, but it does not grant user or administrator +authority. + ## Deletion Lifecycle Lifecycle requests use per-sandbox gates to serialize stop, start, and @@ -457,8 +466,8 @@ watcher emits only sandbox CR changes, not platform events. ### SA Token Authentication -The gateway's `K8sServiceAccountAuthenticator` adapts its `NamespaceValidator` -per mode (`crates/openshell-server/src/auth/k8s_sa.rs`): +The Kubernetes driver's `AuthenticateSandbox` implementation applies its named +`[openshell.drivers.kubernetes]` configuration per mode: - **Shared:** `Exact` — accepts only the single configured namespace. - **Managed:** `Prefix` — accepts any namespace starting with `openshell-{gateway_id}-`. @@ -466,11 +475,14 @@ per mode (`crates/openshell-server/src/auth/k8s_sa.rs`): `BTreeSet` populated by the label/file watchers. Starts empty (fail-closed) until the first watcher update. -These checks rely on an ownership invariant. In shared and managed modes, the -gateway and its trusted Agent Sandbox controller exclusively administer the -sandbox namespace, Sandbox CRs, sandbox pods, and configured sandbox -ServiceAccount. Other principals must not create or mutate those resources or -use that ServiceAccount. In operator mode, the platform operator retains +It validates the projected token with Kubernetes `TokenReview`, checks the live +pod UID, and verifies the pod's controlling Sandbox CR UID and sandbox ID before +returning the identity to the gateway. These checks rely on an ownership +invariant. In shared and managed modes, the Kubernetes driver and its trusted +Agent Sandbox controller exclusively administer the sandbox namespace, Sandbox +CRs, sandbox pods, and configured sandbox ServiceAccount. Other principals must +not create or mutate those resources or use that ServiceAccount. In operator +mode, the platform operator retains namespace lifecycle ownership, but must preserve the same exclusive control of Sandbox CRs and the pods and ServiceAccount used for sandbox token bootstrap. An allowlisted namespace is therefore a trust grant, not a tenant isolation diff --git a/architecture/gateway.md b/architecture/gateway.md index f86f855845..f24da233fa 100644 --- a/architecture/gateway.md +++ b/architecture/gateway.md @@ -225,11 +225,12 @@ identity inspection without client-side token decoding. Sandbox secrets are gateway-signed JWTs bound to a single sandbox ID. Docker, Podman, and VM drivers deliver the initial token through supervisor-only runtime material; Kubernetes supervisors exchange a projected ServiceAccount -token through `IssueSandboxToken`. The gateway validates that projected token -with Kubernetes `TokenReview`, requires the configured sandbox service account, -checks the returned pod binding against the live pod UID, and verifies the pod's -controlling `Sandbox` ownerReference against the live Sandbox CR UID and -sandbox-id label before minting the gateway JWT. The bootstrap path accepts +token through `IssueSandboxToken`. The gateway delegates that opaque credential +to the selected compute driver's `AuthenticateSandbox` RPC. A capable driver is +trusted to return the authenticated sandbox ID, while the gateway still requires +a matching durable sandbox record before minting a JWT. The Kubernetes driver +uses its own named configuration to run TokenReview and verify the live pod and +controlling Sandbox CR. The bootstrap path accepts both `agents.x-k8s.io/v1beta1` ownerReferences from newer Agent Sandbox controllers and `agents.x-k8s.io/v1alpha1` ownerReferences from existing deployments. Supervisors renew gateway JWTs in memory before expiry only while diff --git a/crates/openshell-driver-docker/src/lib.rs b/crates/openshell-driver-docker/src/lib.rs index b1859f54cc..b11c8befac 100644 --- a/crates/openshell-driver-docker/src/lib.rs +++ b/crates/openshell-driver-docker/src/lib.rs @@ -635,6 +635,7 @@ impl DockerComputeDriver { driver_version: self.config.daemon_version.clone(), default_image: self.config.default_image.clone(), gateway_manages_lifecycle: true, + supports_sandbox_authentication: false, } } @@ -1713,6 +1714,19 @@ impl DockerComputeDriver { impl ComputeDriver for ComputeDriverService { type WatchSandboxesStream = WatchStream; + async fn authenticate_sandbox( + &self, + request: Request, + ) -> Result, Status> + { + self.trace_rpc( + "driver.authenticate_sandbox", + "authenticate_sandbox", + ComputeDriver::authenticate_sandbox(&self.driver, request), + ) + .await + } + async fn get_capabilities( &self, request: Request, @@ -1873,6 +1887,16 @@ impl ComputeDriver for ComputeDriverService { #[tonic::async_trait] impl ComputeDriver for DockerComputeDriver { + async fn authenticate_sandbox( + &self, + _request: Request, + ) -> Result, Status> + { + Err(Status::unimplemented( + "docker does not authenticate sandbox credentials", + )) + } + type WatchSandboxesStream = WatchStream; async fn get_capabilities( diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index b352f6efe9..802b5c3110 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -31,7 +31,7 @@ annotation. An OTLP-enabled Agent Sandbox controller can therefore attach its asynchronous reconciliation spans to the originating OpenShell create trace. Workspace namespace modes assume exclusive control of the sandbox identity -resource chain. In shared and managed modes, only the gateway and its trusted +resource chain. In shared and managed modes, only the driver and its trusted Agent Sandbox controller may administer the sandbox namespace, Sandbox CRs, sandbox pods, or configured sandbox ServiceAccount. In operator mode, the platform operator owns namespace lifecycle but must prevent other principals @@ -99,7 +99,9 @@ Sandbox pods run as `service_account_name` and keep `automountServiceAccountToken: false`. The only Kubernetes token exposed to the supervisor is an explicit, audience-bound projected token mounted at `/var/run/secrets/openshell/token` for the one-shot `IssueSandboxToken` -bootstrap exchange. +bootstrap exchange. The Kubernetes driver authenticates that token through the +compute-driver protocol using its own `service_account_name` and workspace-mode +namespace policy; the gateway receives only the verified sandbox ID. The gateway uses the supervisor relay for connect, exec, and file sync. Sandbox pods do not need direct external ingress for SSH. diff --git a/crates/openshell-driver-kubernetes/src/config.rs b/crates/openshell-driver-kubernetes/src/config.rs index 0cf4965011..aadcb1342a 100644 --- a/crates/openshell-driver-kubernetes/src/config.rs +++ b/crates/openshell-driver-kubernetes/src/config.rs @@ -303,7 +303,7 @@ pub struct KubernetesComputeConfig { #[serde(default, skip_serializing_if = "Option::is_none")] pub operator_namespace_file: Option, /// Kubernetes `ServiceAccount` assigned to sandbox pods and accepted by - /// the gateway's `TokenReview` bootstrap authenticator. + /// the driver's `TokenReview` bootstrap authenticator. pub service_account_name: String, pub default_image: String, pub image_pull_policy: String, diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index c1885870e8..64b12c3786 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -8,9 +8,12 @@ use crate::config::{ DEFAULT_PROXY_UID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, DEFAULT_SANDBOX_UID, DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, OperatorNamespaceAllowlist, SupervisorSideloadMethod, SupervisorTopology, WorkspaceMode, is_dns_1123_label, - managed_namespace, validate_managed_namespace_name, + managed_namespace, managed_namespace_prefix, validate_managed_namespace_name, }; use futures::{Stream, StreamExt, TryStreamExt}; +use k8s_openapi::api::authentication::v1::{ + TokenReview, TokenReviewSpec, TokenReviewStatus, UserInfo, +}; use k8s_openapi::api::core::v1::{ Event as KubeEventObj, Namespace, Node, PersistentVolumeClaimVolumeSource, Pod, Secret, ServiceAccount, Volume, VolumeMount, @@ -19,7 +22,7 @@ use k8s_openapi::api::networking::v1::{ NetworkPolicy, NetworkPolicyIngressRule, NetworkPolicyPeer, NetworkPolicyPort, NetworkPolicySpec, }; -use k8s_openapi::apimachinery::pkg::apis::meta::v1::LabelSelector; +use k8s_openapi::apimachinery::pkg::apis::meta::v1::{LabelSelector, OwnerReference}; use k8s_openapi::apimachinery::pkg::util::intstr::IntOrString; use kube::api::{ Api, ApiResource, DeleteParams, ListParams, Patch, PatchParams, PostParams, Preconditions, @@ -118,6 +121,9 @@ pub const SANDBOX_KIND: &str = "Sandbox"; const SANDBOX_POD_NAME_ANNOTATION: &str = "agents.x-k8s.io/pod-name"; const SANDBOX_SUSPENDED_CONDITION: &str = "Suspended"; const SANDBOX_SUSPENDED_POD_NOT_OWNED_REASON: &str = "PodNotOwned"; +const SANDBOX_TOKEN_AUDIENCE: &str = "openshell-gateway"; +const POD_NAME_EXTRA: &str = "authentication.kubernetes.io/pod-name"; +const POD_UID_EXTRA: &str = "authentication.kubernetes.io/pod-uid"; const GPU_RESOURCE_NAME: &str = "nvidia.com/gpu"; const SPIFFE_WORKLOAD_API_VOLUME_NAME: &str = "spiffe-workload-api"; @@ -567,9 +573,71 @@ impl KubernetesComputeDriver { driver_version: openshell_core::VERSION.to_string(), default_image: self.config.default_image.clone(), gateway_manages_lifecycle: false, + supports_sandbox_authentication: true, }) } + /// Authenticate the projected `ServiceAccount` token used by a sandbox pod. + pub async fn authenticate_sandbox(&self, credential: &str) -> Result { + let reviews: Api = Api::all(self.client.clone()); + let review = TokenReview { + metadata: ObjectMeta::default(), + spec: TokenReviewSpec { + audiences: Some(vec![SANDBOX_TOKEN_AUDIENCE.to_string()]), + token: Some(credential.to_string()), + }, + status: None, + }; + let review = reviews + .create(&PostParams::default(), &review) + .await + .map_err(|error| { + warn!(%error, "Kubernetes TokenReview failed"); + tonic::Status::internal("Kubernetes TokenReview failed") + })?; + let status = review + .status + .ok_or_else(|| tonic::Status::internal("TokenReview response missing status"))?; + let identity = token_review_identity(&status, &self.config.service_account_name)? + .ok_or_else(|| tonic::Status::unauthenticated("sandbox credential was not accepted"))?; + if !self.accepts_auth_namespace(&identity.namespace) { + return Err(tonic::Status::permission_denied( + "sandbox credential namespace is not accepted by the driver", + )); + } + + let pods: Api = Api::namespaced(self.client.clone(), &identity.namespace); + let pod = pods + .get_opt(&identity.pod_name) + .await + .map_err(|error| { + warn!(pod = %identity.pod_name, %error, "failed to read authenticated sandbox pod"); + tonic::Status::internal("failed to read authenticated sandbox pod") + })? + .ok_or_else(|| { + tonic::Status::permission_denied("authenticated sandbox pod not found") + })?; + validate_pod_uid(&pod, &identity.pod_uid)?; + let sandbox_id = pod_sandbox_id(&pod)?; + let owner = sandbox_owner_reference(&pod)?; + let sandboxes = self + .supported_agent_sandbox_api(self.client.clone(), &identity.namespace) + .await + .map_err(|error| { + tonic::Status::internal(format!("failed to select Sandbox API: {error}")) + })?; + let sandbox = sandboxes.api.get_opt(&owner.name).await.map_err(|error| { + warn!(sandbox = %owner.name, %error, "failed to read authenticated Sandbox resource"); + tonic::Status::internal("failed to read authenticated Sandbox resource") + })?.ok_or_else(|| tonic::Status::permission_denied("sandbox owner not found"))?; + validate_sandbox_owner_identity(owner, &sandbox_id, &sandbox)?; + Ok(sandbox_id) + } + + fn accepts_auth_namespace(&self, namespace: &str) -> bool { + accepts_auth_namespace(&self.config, self.operator_allowlist.as_ref(), namespace) + } + pub fn operator_allowlist(&self) -> Option<&OperatorNamespaceAllowlist> { self.operator_allowlist.as_ref() } @@ -2232,6 +2300,161 @@ fn sandbox_id_from_object(obj: &DynamicObject) -> Result { Err("sandbox id not found on object".to_string()) } +#[derive(Debug)] +struct TokenReviewIdentity { + namespace: String, + pod_name: String, + pod_uid: String, +} + +#[allow(clippy::result_large_err)] +fn token_review_identity( + status: &TokenReviewStatus, + expected_service_account: &str, +) -> Result, tonic::Status> { + if status.authenticated != Some(true) { + return Ok(None); + } + if !status + .audiences + .as_deref() + .unwrap_or_default() + .iter() + .any(|audience| audience == SANDBOX_TOKEN_AUDIENCE) + { + return Err(tonic::Status::unauthenticated( + "sandbox credential audience not accepted", + )); + } + let user = status + .user + .as_ref() + .ok_or_else(|| tonic::Status::permission_denied("TokenReview response missing user"))?; + let rest = user + .username + .as_deref() + .unwrap_or_default() + .strip_prefix("system:serviceaccount:") + .ok_or_else(|| tonic::Status::permission_denied("credential is not a service account"))?; + let (namespace, service_account) = rest + .split_once(':') + .filter(|(namespace, service_account)| !namespace.is_empty() && !service_account.is_empty()) + .ok_or_else(|| tonic::Status::permission_denied("invalid service account identity"))?; + if service_account != expected_service_account { + return Err(tonic::Status::permission_denied( + "credential is not from the configured sandbox service account", + )); + } + Ok(Some(TokenReviewIdentity { + namespace: namespace.to_string(), + pod_name: user_extra_one(user, POD_NAME_EXTRA)?, + pod_uid: user_extra_one(user, POD_UID_EXTRA)?, + })) +} + +#[allow(clippy::result_large_err)] +fn user_extra_one(user: &UserInfo, key: &str) -> Result { + let values = user + .extra + .as_ref() + .and_then(|extra| extra.get(key)) + .ok_or_else(|| tonic::Status::permission_denied("sandbox credential is not pod-bound"))?; + if values.len() != 1 || values[0].is_empty() { + return Err(tonic::Status::permission_denied( + "sandbox credential has invalid pod binding", + )); + } + Ok(values[0].clone()) +} + +#[allow(clippy::result_large_err)] +fn pod_sandbox_id(pod: &Pod) -> Result { + pod.metadata + .annotations + .as_ref() + .and_then(|annotations| annotations.get(LABEL_SANDBOX_ID)) + .filter(|value| !value.is_empty()) + .cloned() + .ok_or_else(|| tonic::Status::permission_denied("pod is not bound to a sandbox identity")) +} + +#[allow(clippy::result_large_err)] +fn validate_pod_uid(pod: &Pod, expected_uid: &str) -> Result<(), tonic::Status> { + if pod.metadata.uid.as_deref() == Some(expected_uid) { + return Ok(()); + } + Err(tonic::Status::permission_denied( + "sandbox credential pod UID mismatch", + )) +} + +#[allow(clippy::result_large_err)] +fn sandbox_owner_reference(pod: &Pod) -> Result<&OwnerReference, tonic::Status> { + let mut owners = pod + .metadata + .owner_references + .as_deref() + .unwrap_or_default() + .iter() + .filter(|owner| { + owner.kind == SANDBOX_KIND + && matches!( + owner.api_version.as_str(), + "agents.x-k8s.io/v1beta1" | "agents.x-k8s.io/v1alpha1" + ) + }); + let owner = owners + .next() + .ok_or_else(|| tonic::Status::permission_denied("pod is not controlled by a Sandbox"))?; + if owners.next().is_some() + || owner.controller != Some(true) + || owner.name.is_empty() + || owner.uid.is_empty() + { + return Err(tonic::Status::permission_denied( + "pod has an invalid Sandbox owner", + )); + } + Ok(owner) +} + +#[allow(clippy::result_large_err)] +fn validate_sandbox_owner_identity( + owner: &OwnerReference, + sandbox_id: &str, + sandbox: &DynamicObject, +) -> Result<(), tonic::Status> { + let uid_matches = sandbox.metadata.uid.as_deref() == Some(owner.uid.as_str()); + let sandbox_id_matches = sandbox + .metadata + .labels + .as_ref() + .and_then(|labels| labels.get(LABEL_SANDBOX_ID)) + .is_some_and(|actual| actual == sandbox_id); + if uid_matches && sandbox_id_matches { + return Ok(()); + } + Err(tonic::Status::permission_denied( + "pod identity does not match its Sandbox owner", + )) +} + +fn accepts_auth_namespace( + config: &KubernetesComputeConfig, + operator_allowlist: Option<&OperatorNamespaceAllowlist>, + namespace: &str, +) -> bool { + match config.workspace_mode { + WorkspaceMode::Shared => namespace == config.namespace, + WorkspaceMode::Managed => { + namespace.starts_with(&managed_namespace_prefix(&config.gateway_id)) + } + WorkspaceMode::Operator => { + operator_allowlist.is_some_and(|allowlist| allowlist.contains(namespace)) + } + } +} + fn annotation_or_label(obj: &DynamicObject, key: &str) -> Option { obj.metadata .annotations @@ -3558,7 +3781,7 @@ fn sandbox_template_to_k8s_with_validated_config( .unwrap_or_default(); if !params.sandbox_id.is_empty() { pod_annotations.insert( - "openshell.io/sandbox-id".to_string(), + LABEL_SANDBOX_ID.to_string(), serde_json::Value::String(params.sandbox_id.to_string()), ); } @@ -4644,6 +4867,7 @@ mod tests { }; use openshell_core::proto::compute::v1::{GpuResourceRequirements, ResourceRequirements}; use prost_types::{Struct, Value, value::Kind}; + use std::collections::BTreeSet; static ENV_LOCK: std::sync::LazyLock> = std::sync::LazyLock::new(|| std::sync::Mutex::new(())); @@ -4744,6 +4968,180 @@ mod tests { }) } + fn authenticated_token_review(username: &str) -> TokenReviewStatus { + TokenReviewStatus { + authenticated: Some(true), + audiences: Some(vec![SANDBOX_TOKEN_AUDIENCE.to_string()]), + user: Some(UserInfo { + username: Some(username.to_string()), + extra: Some(BTreeMap::from([ + (POD_NAME_EXTRA.to_string(), vec!["sandbox-pod".to_string()]), + (POD_UID_EXTRA.to_string(), vec!["pod-uid".to_string()]), + ])), + ..Default::default() + }), + ..Default::default() + } + } + + #[test] + fn token_review_uses_configured_service_account_and_pod_binding() { + let status = authenticated_token_review("system:serviceaccount:workspaces:sandbox-sa"); + let identity = token_review_identity(&status, "sandbox-sa") + .unwrap() + .expect("authenticated identity"); + assert_eq!(identity.namespace, "workspaces"); + assert_eq!(identity.pod_name, "sandbox-pod"); + assert_eq!(identity.pod_uid, "pod-uid"); + } + + #[test] + fn token_review_rejects_a_different_service_account() { + let status = authenticated_token_review("system:serviceaccount:workspaces:other"); + let error = token_review_identity(&status, "sandbox-sa").unwrap_err(); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + } + + #[test] + fn token_review_rejects_wrong_audience_and_missing_pod_binding() { + let mut wrong_audience = + authenticated_token_review("system:serviceaccount:workspaces:sandbox-sa"); + wrong_audience.audiences = Some(vec!["kubernetes.default.svc".to_string()]); + let error = token_review_identity(&wrong_audience, "sandbox-sa").unwrap_err(); + assert_eq!(error.code(), tonic::Code::Unauthenticated); + + let mut missing_binding = + authenticated_token_review("system:serviceaccount:workspaces:sandbox-sa"); + missing_binding.user.as_mut().unwrap().extra = None; + let error = token_review_identity(&missing_binding, "sandbox-sa").unwrap_err(); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + } + + #[test] + fn token_review_returns_none_when_not_authenticated() { + let status = TokenReviewStatus { + authenticated: Some(false), + error: Some("token rejected".to_string()), + ..Default::default() + }; + + assert!( + token_review_identity(&status, "sandbox-sa") + .unwrap() + .is_none() + ); + } + + #[test] + fn authentication_namespace_validation_covers_each_workspace_mode() { + let mut config = KubernetesComputeConfig { + namespace: "openshell".to_string(), + ..Default::default() + }; + assert!(accepts_auth_namespace(&config, None, "openshell")); + assert!(!accepts_auth_namespace(&config, None, "other")); + + config.workspace_mode = WorkspaceMode::Managed; + config.gateway_id = "gateway-a".to_string(); + assert!(accepts_auth_namespace( + &config, + None, + "openshell-gateway-a-workspace-a" + )); + assert!(!accepts_auth_namespace( + &config, + None, + "openshell-gateway-b-workspace-a" + )); + + config.workspace_mode = WorkspaceMode::Operator; + let allowlist = OperatorNamespaceAllowlist::from_set(BTreeSet::from([ + "team-a".to_string(), + "team-b".to_string(), + ])); + assert!(accepts_auth_namespace(&config, Some(&allowlist), "team-a")); + assert!(!accepts_auth_namespace(&config, Some(&allowlist), "team-c")); + assert!(!accepts_auth_namespace(&config, None, "team-a")); + } + + fn sandbox_owner_for_test(name: &str, uid: &str) -> OwnerReference { + OwnerReference { + api_version: "agents.x-k8s.io/v1beta1".to_string(), + block_owner_deletion: None, + controller: Some(true), + kind: SANDBOX_KIND.to_string(), + name: name.to_string(), + uid: uid.to_string(), + } + } + + fn sandbox_object_for_test(uid: &str, sandbox_id: &str) -> DynamicObject { + let resource = ApiResource::from_gvk(&GroupVersionKind::gvk( + SANDBOX_GROUP, + SANDBOX_VERSION_V1BETA1, + SANDBOX_KIND, + )); + let mut sandbox = DynamicObject::new("sandbox-a", &resource); + sandbox.metadata.uid = Some(uid.to_string()); + sandbox.metadata.labels = Some(BTreeMap::from([( + LABEL_SANDBOX_ID.to_string(), + sandbox_id.to_string(), + )])); + sandbox + } + + #[test] + fn pod_identity_requires_matching_uid_annotation_and_controlling_owner() { + let owner = sandbox_owner_for_test("sandbox-a", "sandbox-uid-a"); + let pod = Pod { + metadata: ObjectMeta { + uid: Some("pod-uid-a".to_string()), + annotations: Some(BTreeMap::from([( + LABEL_SANDBOX_ID.to_string(), + "sandbox-id-a".to_string(), + )])), + owner_references: Some(vec![owner.clone()]), + ..Default::default() + }, + ..Default::default() + }; + + validate_pod_uid(&pod, "pod-uid-a").expect("matching pod UID"); + assert_eq!(pod_sandbox_id(&pod).unwrap(), "sandbox-id-a"); + assert_eq!(sandbox_owner_reference(&pod).unwrap(), &owner); + + let error = validate_pod_uid(&pod, "other-pod-uid").unwrap_err(); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + + let mut missing_annotation = pod.clone(); + missing_annotation.metadata.annotations = None; + let error = pod_sandbox_id(&missing_annotation).unwrap_err(); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + + let mut non_controlling = pod; + non_controlling.metadata.owner_references.as_mut().unwrap()[0].controller = Some(false); + let error = sandbox_owner_reference(&non_controlling).unwrap_err(); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + } + + #[test] + fn sandbox_owner_identity_requires_matching_uid_and_sandbox_id() { + let owner = sandbox_owner_for_test("sandbox-a", "sandbox-uid-a"); + let sandbox = sandbox_object_for_test("sandbox-uid-a", "sandbox-id-a"); + validate_sandbox_owner_identity(&owner, "sandbox-id-a", &sandbox) + .expect("matching owner identity"); + + let mismatched_owner = sandbox_object_for_test("sandbox-uid-b", "sandbox-id-a"); + let error = + validate_sandbox_owner_identity(&owner, "sandbox-id-a", &mismatched_owner).unwrap_err(); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + + let mismatched_annotation = sandbox_object_for_test("sandbox-uid-a", "sandbox-id-b"); + let error = validate_sandbox_owner_identity(&owner, "sandbox-id-a", &mismatched_annotation) + .unwrap_err(); + assert_eq!(error.code(), tonic::Code::PermissionDenied); + } + #[test] fn sandbox_api_version_probe_retries_on_structured_and_raw_404() { let structured = kube_api_error(404, "could not find the requested resource"); @@ -7120,6 +7518,24 @@ mod tests { ); } + #[test] + fn sandbox_template_annotation_is_accepted_by_bootstrap_authentication() { + let params = SandboxPodParams { + sandbox_id: "sandbox-a", + ..Default::default() + }; + let pod_template = sandbox_template_to_k8s( + &SandboxTemplate::default(), + false, + &std::collections::HashMap::new(), + true, + ¶ms, + ); + let pod: Pod = serde_json::from_value(pod_template).expect("valid pod template"); + + assert_eq!(pod_sandbox_id(&pod).unwrap(), "sandbox-a"); + } + #[test] fn sandbox_template_omits_empty_image_pull_secrets() { let pod_template = sandbox_template_to_k8s( diff --git a/crates/openshell-driver-kubernetes/src/grpc.rs b/crates/openshell-driver-kubernetes/src/grpc.rs index 383ddebe0c..027c350a26 100644 --- a/crates/openshell-driver-kubernetes/src/grpc.rs +++ b/crates/openshell-driver-kubernetes/src/grpc.rs @@ -5,14 +5,15 @@ use futures::{Stream, StreamExt}; use openshell_core::proto::compute::v1::{ - CreateSandboxRequest, CreateSandboxResponse, DeleteSandboxRequest, DeleteSandboxResponse, - DeleteWorkspaceRequest, DeleteWorkspaceResponse, EnsureWorkspaceRequest, - EnsureWorkspaceResponse, GetCapabilitiesRequest, GetCapabilitiesResponse, - GetGatewayListenerRequirementsRequest, GetGatewayListenerRequirementsResponse, - GetSandboxRequest, GetSandboxResponse, ListSandboxesRequest, ListSandboxesResponse, - StartSandboxRequest, StartSandboxResponse, StopSandboxRequest, StopSandboxResponse, - ValidateSandboxCreateRequest, ValidateSandboxCreateResponse, WatchSandboxesEvent, - WatchSandboxesRequest, compute_driver_server::ComputeDriver, + AuthenticateSandboxRequest, AuthenticateSandboxResponse, CreateSandboxRequest, + CreateSandboxResponse, DeleteSandboxRequest, DeleteSandboxResponse, DeleteWorkspaceRequest, + DeleteWorkspaceResponse, EnsureWorkspaceRequest, EnsureWorkspaceResponse, + GetCapabilitiesRequest, GetCapabilitiesResponse, GetGatewayListenerRequirementsRequest, + GetGatewayListenerRequirementsResponse, GetSandboxRequest, GetSandboxResponse, + ListSandboxesRequest, ListSandboxesResponse, StartSandboxRequest, StartSandboxResponse, + StopSandboxRequest, StopSandboxResponse, ValidateSandboxCreateRequest, + ValidateSandboxCreateResponse, WatchSandboxesEvent, WatchSandboxesRequest, + compute_driver_server::ComputeDriver, }; use std::future::Future; use std::pin::Pin; @@ -122,6 +123,18 @@ impl ComputeDriverService { #[tonic::async_trait] impl ComputeDriver for ComputeDriverService { + async fn authenticate_sandbox( + &self, + request: Request, + ) -> Result, Status> { + let credential = request.into_inner().credential; + if credential.is_empty() { + return Err(Status::invalid_argument("credential is required")); + } + let sandbox_id = self.driver.authenticate_sandbox(&credential).await?; + Ok(Response::new(AuthenticateSandboxResponse { sandbox_id })) + } + async fn get_capabilities( &self, _request: Request, diff --git a/crates/openshell-driver-podman/src/driver.rs b/crates/openshell-driver-podman/src/driver.rs index a11189cbc6..8f7c0d32f6 100644 --- a/crates/openshell-driver-podman/src/driver.rs +++ b/crates/openshell-driver-podman/src/driver.rs @@ -507,6 +507,7 @@ impl PodmanComputeDriver { driver_version: openshell_core::VERSION.to_string(), default_image: self.config.default_image.clone(), gateway_manages_lifecycle: true, + supports_sandbox_authentication: false, }) } diff --git a/crates/openshell-driver-podman/src/grpc.rs b/crates/openshell-driver-podman/src/grpc.rs index 2cb59f086f..222c617c2f 100644 --- a/crates/openshell-driver-podman/src/grpc.rs +++ b/crates/openshell-driver-podman/src/grpc.rs @@ -127,6 +127,16 @@ impl ComputeDriverService { #[tonic::async_trait] impl ComputeDriver for ComputeDriverService { + async fn authenticate_sandbox( + &self, + _request: Request, + ) -> Result, Status> + { + Err(Status::unimplemented( + "podman does not authenticate sandbox credentials", + )) + } + async fn get_capabilities( &self, _request: Request, diff --git a/crates/openshell-driver-vm/src/driver.rs b/crates/openshell-driver-vm/src/driver.rs index 13e57f546d..2de65c3add 100644 --- a/crates/openshell-driver-vm/src/driver.rs +++ b/crates/openshell-driver-vm/src/driver.rs @@ -524,6 +524,7 @@ impl VmDriver { driver_version: openshell_core::VERSION.to_string(), default_image: self.config.default_image.clone(), gateway_manages_lifecycle: true, + supports_sandbox_authentication: false, } } @@ -3325,6 +3326,16 @@ impl VmDriver { #[tonic::async_trait] impl ComputeDriver for VmDriver { + async fn authenticate_sandbox( + &self, + _request: Request, + ) -> Result, Status> + { + Err(Status::unimplemented( + "VM driver does not authenticate sandbox credentials", + )) + } + async fn get_capabilities( &self, _request: Request, diff --git a/crates/openshell-server/src/auth/authenticator.rs b/crates/openshell-server/src/auth/authenticator.rs index f5d5c7b2af..5511ef79db 100644 --- a/crates/openshell-server/src/auth/authenticator.rs +++ b/crates/openshell-server/src/auth/authenticator.rs @@ -14,8 +14,8 @@ //! //! Live authenticators slotting into the chain: //! - [`super::sandbox_jwt::SandboxJwtAuthenticator`] — gateway-minted JWTs -//! - [`super::k8s_sa::K8sServiceAccountAuthenticator`] — K8s projected SA -//! tokens (path-scoped to `IssueSandboxToken`) +//! - [`super::compute_driver::ComputeDriverAuthenticator`] — driver-native +//! sandbox bootstrap credentials (path-scoped to `IssueSandboxToken`) //! - [`super::oidc::OidcAuthenticator`] — user OIDC Bearer tokens use super::principal::Principal; use async_trait::async_trait; diff --git a/crates/openshell-server/src/auth/compute_driver.rs b/crates/openshell-server/src/auth/compute_driver.rs new file mode 100644 index 0000000000..cedc2115b5 --- /dev/null +++ b/crates/openshell-server/src/auth/compute_driver.rs @@ -0,0 +1,176 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Compute-driver delegated sandbox bootstrap authentication. + +use super::authenticator::Authenticator; +use super::principal::{Principal, SandboxIdentitySource, SandboxPrincipal}; +use crate::compute::ComputeRuntime; +use async_trait::async_trait; +use tonic::Status; + +/// The only public gateway method on which driver-native credentials apply. +pub const ISSUE_SANDBOX_TOKEN_PATH: &str = "/openshell.v1.OpenShell/IssueSandboxToken"; + +#[derive(Clone, Debug)] +pub struct ComputeDriverAuthenticator { + compute: ComputeRuntime, +} + +impl ComputeDriverAuthenticator { + pub fn new(compute: ComputeRuntime) -> Self { + Self { compute } + } +} + +#[async_trait] +impl Authenticator for ComputeDriverAuthenticator { + async fn authenticate( + &self, + headers: &http::HeaderMap, + path: &str, + ) -> Result, Status> { + if path != ISSUE_SANDBOX_TOKEN_PATH { + return Ok(None); + } + + let Some(credential) = headers + .get(http::header::AUTHORIZATION) + .and_then(|value| value.to_str().ok()) + .and_then(|value| value.strip_prefix("Bearer ")) + else { + return Ok(None); + }; + + let sandbox_id = self.compute.authenticate_sandbox(credential).await?; + if sandbox_id.is_empty() { + return Err(Status::permission_denied( + "compute driver returned an empty sandbox identity", + )); + } + + Ok(Some(Principal::Sandbox(SandboxPrincipal { + sandbox_id, + source: SandboxIdentitySource::ComputeDriver { + driver_name: self.compute.selected_driver_name().to_string(), + }, + trust_domain: Some("openshell".to_string()), + }))) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::auth::principal::SandboxIdentitySource; + use crate::compute::{NoopTestDriver, new_test_runtime_with_driver}; + use crate::persistence::Store; + use std::sync::Arc; + use tonic::Code; + + fn bearer_headers(token: &str) -> http::HeaderMap { + let mut headers = http::HeaderMap::new(); + headers.insert( + http::header::AUTHORIZATION, + http::HeaderValue::from_str(&format!("Bearer {token}")).unwrap(), + ); + headers + } + + async fn authenticator(driver: NoopTestDriver) -> ComputeDriverAuthenticator { + let store = Arc::new(Store::connect("sqlite::memory:").await.unwrap()); + let compute = + new_test_runtime_with_driver(store, "external-kubernetes", Arc::new(driver)).await; + ComputeDriverAuthenticator::new(compute) + } + + #[tokio::test] + async fn authenticates_driver_credential_on_issue_path() { + let auth = authenticator(NoopTestDriver::authenticating_sandbox("sandbox-a")).await; + + let principal = auth + .authenticate( + &bearer_headers("driver-credential"), + ISSUE_SANDBOX_TOKEN_PATH, + ) + .await + .unwrap() + .expect("driver credential should authenticate"); + + let Principal::Sandbox(principal) = principal else { + panic!("expected sandbox principal"); + }; + assert_eq!(principal.sandbox_id, "sandbox-a"); + assert!(matches!( + principal.source, + SandboxIdentitySource::ComputeDriver { ref driver_name } + if driver_name == "external-kubernetes" + )); + } + + #[tokio::test] + async fn authenticator_is_scoped_to_issue_path() { + let auth = authenticator(NoopTestDriver::failing_sandbox_authentication( + Code::Unavailable, + "driver must not be called", + )) + .await; + + let result = auth + .authenticate( + &bearer_headers("driver-credential"), + "/openshell.v1.OpenShell/GetSandboxConfig", + ) + .await + .unwrap(); + + assert!(result.is_none()); + } + + #[tokio::test] + async fn missing_bearer_credential_falls_through() { + let auth = authenticator(NoopTestDriver::authenticating_sandbox("sandbox-a")).await; + + let result = auth + .authenticate(&http::HeaderMap::new(), ISSUE_SANDBOX_TOKEN_PATH) + .await + .unwrap(); + + assert!(result.is_none()); + } + + #[tokio::test] + async fn empty_driver_identity_is_rejected() { + let auth = authenticator(NoopTestDriver::authenticating_sandbox("")).await; + + let error = auth + .authenticate( + &bearer_headers("driver-credential"), + ISSUE_SANDBOX_TOKEN_PATH, + ) + .await + .expect_err("empty identity must fail closed"); + + assert_eq!(error.code(), Code::PermissionDenied); + } + + #[tokio::test] + async fn driver_authentication_error_propagates() { + let auth = authenticator(NoopTestDriver::failing_sandbox_authentication( + Code::Unavailable, + "driver unavailable", + )) + .await; + + let error = auth + .authenticate( + &bearer_headers("driver-credential"), + ISSUE_SANDBOX_TOKEN_PATH, + ) + .await + .expect_err("driver errors must propagate"); + + assert_eq!(error.code(), Code::Unavailable); + assert_eq!(error.message(), "driver unavailable"); + } +} diff --git a/crates/openshell-server/src/auth/k8s_sa.rs b/crates/openshell-server/src/auth/k8s_sa.rs deleted file mode 100644 index 131dbaba47..0000000000 --- a/crates/openshell-server/src/auth/k8s_sa.rs +++ /dev/null @@ -1,1098 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Kubernetes `ServiceAccount` bootstrap authenticator. -//! -//! Path-scoped to `IssueSandboxToken`. Validates a projected SA token -//! presented by a sandbox pod, reads the pod's `openshell.io/sandbox-id` -//! annotation, verifies the pod is controlled by the corresponding Sandbox CR, -//! and returns a [`Principal::Sandbox`] with -//! [`SandboxIdentitySource::K8sServiceAccount`]. The `IssueSandboxToken` handler -//! then mints a gateway-signed JWT for that sandbox id; subsequent gRPC calls -//! from the supervisor use the gateway-minted JWT validated by -//! [`super::sandbox_jwt::SandboxJwtAuthenticator`]. -//! -//! This is the only authenticator that talks to the K8s apiserver. It is -//! optional — the gateway boots without it in singleplayer deployments. - -use super::authenticator::Authenticator; -use super::principal::{Principal, SandboxIdentitySource, SandboxPrincipal}; -use async_trait::async_trait; -use k8s_openapi::api::{ - authentication::v1::{TokenReview, TokenReviewSpec, TokenReviewStatus, UserInfo}, - core::v1::Pod, -}; -use k8s_openapi::apimachinery::pkg::apis::meta::v1::ObjectMeta; -use kube::Error as KubeError; -use kube::api::{Api, ApiResource, PostParams}; -use kube::core::{DynamicObject, gvk::GroupVersionKind}; -use openshell_core::OperatorNamespaceAllowlist; -use std::sync::Arc; -use tonic::Status; -use tracing::{debug, info, warn}; - -/// gRPC method path that this authenticator accepts. All other paths fall -/// through (return `Ok(None)`) so a gateway-minted JWT is required there. -pub const ISSUE_SANDBOX_TOKEN_PATH: &str = "/openshell.v1.OpenShell/IssueSandboxToken"; - -/// Pod annotation that binds a sandbox pod to its UUID. Set by the -/// Kubernetes compute driver at pod-create time. The gateway accepts this -/// annotation only after validating the pod's `TokenReview` binding, live UID, -/// and owning Sandbox CR. The K8s `Role` granted to the gateway must not -/// include `patch pods` (see plan §11.8). -pub const SANDBOX_ID_ANNOTATION: &str = "openshell.io/sandbox-id"; -const SANDBOX_API_GROUP: &str = "agents.x-k8s.io"; -const SANDBOX_API_VERSION_V1BETA1: &str = "v1beta1"; -const SANDBOX_API_VERSION_V1ALPHA1: &str = "v1alpha1"; -const SANDBOX_API_VERSION_FULL_V1BETA1: &str = "agents.x-k8s.io/v1beta1"; -const SANDBOX_API_VERSION_FULL_V1ALPHA1: &str = "agents.x-k8s.io/v1alpha1"; -const SANDBOX_KIND: &str = "Sandbox"; -const SANDBOX_ID_LABEL: &str = "openshell.ai/sandbox-id"; -const POD_NAME_EXTRA: &str = "authentication.kubernetes.io/pod-name"; -const POD_UID_EXTRA: &str = "authentication.kubernetes.io/pod-uid"; - -/// Resolved identity extracted from a validated SA token + pod lookup. -#[derive(Debug, Clone)] -pub struct ResolvedK8sIdentity { - pub sandbox_id: String, - pub pod_name: String, - pub pod_uid: String, -} - -/// Apiserver-facing operations the authenticator depends on. Split out so -/// tests can fake the apiserver without standing up a kube cluster. -#[async_trait] -pub trait K8sIdentityResolver: Send + Sync + 'static { - /// Validate `token` via `TokenReview` (`aud == openshell-gateway`), - /// extract the pod name/uid, then `GET` the pod and read - /// `openshell.io/sandbox-id`. Returns `Ok(None)` when the token is - /// well-formed but does not authenticate (e.g. wrong audience); returns - /// `Err` for transport/server errors. - async fn resolve(&self, token: &str) -> Result, Status>; -} - -/// Authenticator wrapper around a [`K8sIdentityResolver`]. -pub struct K8sServiceAccountAuthenticator { - resolver: Arc, -} - -impl std::fmt::Debug for K8sServiceAccountAuthenticator { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.debug_struct("K8sServiceAccountAuthenticator") - .finish_non_exhaustive() - } -} - -impl K8sServiceAccountAuthenticator { - pub fn new(resolver: Arc) -> Self { - Self { resolver } - } -} - -#[async_trait] -impl Authenticator for K8sServiceAccountAuthenticator { - async fn authenticate( - &self, - headers: &http::HeaderMap, - path: &str, - ) -> Result, Status> { - // Scope: only the bootstrap RPC. Other paths fall through so the - // SandboxJwtAuthenticator (or OIDC) handles them. - if path != ISSUE_SANDBOX_TOKEN_PATH { - return Ok(None); - } - - let Some(token) = headers - .get("authorization") - .and_then(|v| v.to_str().ok()) - .and_then(|v| v.strip_prefix("Bearer ")) - else { - return Ok(None); - }; - - let Some(resolved) = self.resolver.resolve(token).await? else { - debug!("K8s SA token did not authenticate; falling through"); - return Ok(None); - }; - - if resolved.sandbox_id.is_empty() { - warn!( - pod = %resolved.pod_name, - "pod missing openshell.io/sandbox-id annotation; rejecting" - ); - return Err(Status::permission_denied( - "pod is not bound to a sandbox identity", - )); - } - - Ok(Some(Principal::Sandbox(SandboxPrincipal { - sandbox_id: resolved.sandbox_id, - source: SandboxIdentitySource::K8sServiceAccount { - pod_name: resolved.pod_name, - pod_uid: resolved.pod_uid, - }, - trust_domain: Some("openshell".to_string()), - }))) - } -} - -/// Validates the namespace extracted from an SA token username against the -/// expected set for the active workspace mode. -#[derive(Debug, Clone)] -pub enum NamespaceValidator { - /// Shared mode: accept only the single configured namespace. - Exact(String), - /// Managed mode: accept any namespace with the managed prefix - /// (`openshell-{gateway_id}-`). - Prefix(String), - /// Operator mode: accept namespaces in the dynamic allowlist. - Allowlist(OperatorNamespaceAllowlist), -} - -impl NamespaceValidator { - pub fn accepts(&self, namespace: &str) -> bool { - match self { - Self::Exact(expected) => namespace == expected, - Self::Prefix(prefix) => namespace.starts_with(prefix.as_str()), - Self::Allowlist(al) => al.contains(namespace), - } - } -} - -#[derive(Debug)] -struct TokenReviewIdentity { - namespace: String, - pod_name: String, - pod_uid: String, -} - -#[derive(Debug, Clone, PartialEq, Eq)] -struct SandboxOwnerReference { - api_version: String, - name: String, - uid: String, -} - -/// Resolver backed by the apiserver's `TokenReview` API and `kube::Client` -/// for the per-pod annotation lookup. -pub struct LiveK8sResolver { - client: kube::Client, - token_reviews_api: Api, - expected_audience: String, - namespace_validator: NamespaceValidator, - expected_service_account: String, -} - -impl LiveK8sResolver { - pub fn new( - client: kube::Client, - namespace_validator: NamespaceValidator, - expected_audience: String, - expected_service_account: String, - ) -> Self { - let token_reviews_api: Api = Api::all(client.clone()); - Self { - client, - token_reviews_api, - expected_audience, - namespace_validator, - expected_service_account, - } - } - - fn pods_api(&self, namespace: &str) -> Api { - Api::namespaced(self.client.clone(), namespace) - } - - fn sandboxes_api(&self, namespace: &str, api_version: &str) -> Api { - let gvk = GroupVersionKind::gvk(SANDBOX_API_GROUP, api_version, SANDBOX_KIND); - let resource = ApiResource::from_gvk(&gvk); - Api::namespaced_with(self.client.clone(), namespace, &resource) - } - - async fn get_sandbox_cr_for_owner( - &self, - namespace: &str, - owner: &SandboxOwnerReference, - ) -> Result, KubeError> { - let versions = if owner.api_version == SANDBOX_API_VERSION_FULL_V1ALPHA1 { - [SANDBOX_API_VERSION_V1ALPHA1, SANDBOX_API_VERSION_V1BETA1] - } else { - [SANDBOX_API_VERSION_V1BETA1, SANDBOX_API_VERSION_V1ALPHA1] - }; - - for version in versions { - let api = self.sandboxes_api(namespace, version); - match api.get_opt(&owner.name).await { - Ok(Some(sandbox_cr)) => return Ok(Some(sandbox_cr)), - Ok(None) => {} - Err(err) if should_try_next_sandbox_api_version(&err) => {} - Err(err) => return Err(err), - } - } - - Ok(None) - } -} - -#[async_trait] -impl K8sIdentityResolver for LiveK8sResolver { - async fn resolve(&self, token: &str) -> Result, Status> { - let review = TokenReview { - metadata: ObjectMeta::default(), - spec: TokenReviewSpec { - audiences: Some(vec![self.expected_audience.clone()]), - token: Some(token.to_string()), - }, - status: None, - }; - - let review = self - .token_reviews_api - .create(&PostParams::default(), &review) - .await - .map_err(|e| { - warn!(error = %e, "K8s TokenReview failed"); - Status::internal(format!("tokenreview failed: {e}")) - })?; - let status = review - .status - .ok_or_else(|| Status::internal("TokenReview response missing status"))?; - let Some(identity) = token_review_identity( - &status, - &self.expected_audience, - &self.namespace_validator, - &self.expected_service_account, - )? - else { - return Ok(None); - }; - - info!( - pod_name = %identity.pod_name, - pod_uid = %identity.pod_uid, - namespace = %identity.namespace, - service_account = %self.expected_service_account, - "validated K8s SA token via TokenReview" - ); - - let pods_api = self.pods_api(&identity.namespace); - let pod = pods_api.get_opt(&identity.pod_name).await.map_err(|e| { - warn!( - pod = %identity.pod_name, - namespace = %identity.namespace, - error = %e, - "failed to fetch sandbox pod for annotation lookup" - ); - Status::internal(format!("pod GET failed: {e}")) - })?; - let Some(pod) = pod else { - warn!( - pod = %identity.pod_name, - namespace = %identity.namespace, - "sandbox pod referenced by SA token not found" - ); - return Err(Status::not_found("sandbox pod not found")); - }; - - let actual_uid = pod.metadata.uid.as_deref().unwrap_or_default(); - if actual_uid != identity.pod_uid { - warn!( - pod = %identity.pod_name, - claimed_uid = %identity.pod_uid, - actual_uid = %actual_uid, - "SA token pod UID does not match live pod; rejecting" - ); - return Err(Status::permission_denied("SA token pod UID mismatch")); - } - - let sandbox_id = pod_sandbox_id(&pod)?; - - let owner = sandbox_owner_reference(&pod)?; - let sandbox_cr = self - .get_sandbox_cr_for_owner(&identity.namespace, &owner) - .await - .map_err(|e| { - warn!( - pod = %identity.pod_name, - sandbox_owner = %owner.name, - sandbox_owner_api_version = %owner.api_version, - error = %e, - "failed to fetch owning Sandbox CR for pod identity validation" - ); - Status::internal(format!("sandbox GET failed: {e}")) - })?; - let Some(sandbox_cr) = sandbox_cr else { - warn!( - pod = %identity.pod_name, - sandbox_owner = %owner.name, - sandbox_owner_api_version = %owner.api_version, - "pod ownerReference points to a Sandbox CR that does not exist" - ); - return Err(Status::permission_denied("sandbox owner not found")); - }; - validate_sandbox_owner_reference(&owner, &sandbox_id, &sandbox_cr)?; - - Ok(Some(ResolvedK8sIdentity { - sandbox_id, - pod_name: identity.pod_name, - pod_uid: identity.pod_uid, - })) - } -} - -#[allow(clippy::result_large_err)] -fn token_review_identity( - status: &TokenReviewStatus, - expected_audience: &str, - namespace_validator: &NamespaceValidator, - expected_service_account: &str, -) -> Result, Status> { - if status.authenticated != Some(true) { - debug!( - error = status.error.as_deref().unwrap_or_default(), - "K8s TokenReview did not authenticate token" - ); - return Ok(None); - } - - let audiences = status.audiences.as_deref().unwrap_or_default(); - if !audiences.iter().any(|aud| aud == expected_audience) { - warn!( - expected_audience = %expected_audience, - audiences = ?audiences, - "K8s TokenReview authenticated token without expected audience" - ); - return Err(Status::unauthenticated("SA token audience not accepted")); - } - - let user = status - .user - .as_ref() - .ok_or_else(|| Status::permission_denied("TokenReview response missing user info"))?; - let username = user - .username - .as_deref() - .ok_or_else(|| Status::permission_denied("TokenReview response missing username"))?; - - let (namespace, sa_name) = parse_sa_username(username).ok_or_else(|| { - warn!( - username = %username, - "K8s TokenReview username is not a service account" - ); - Status::permission_denied("SA token username format not recognized") - })?; - - if sa_name != expected_service_account { - warn!( - username = %username, - service_account = %sa_name, - expected = %expected_service_account, - "K8s TokenReview principal is not the configured sandbox service account" - ); - return Err(Status::permission_denied( - "SA token is not from the configured sandbox service account", - )); - } - - if !namespace_validator.accepts(&namespace) { - warn!( - username = %username, - namespace = %namespace, - "K8s TokenReview SA namespace not accepted by workspace mode validator" - ); - return Err(Status::permission_denied( - "SA token is not from an accepted sandbox namespace", - )); - } - - let pod_name = user_extra_one(user, POD_NAME_EXTRA)?; - let pod_uid = user_extra_one(user, POD_UID_EXTRA)?; - Ok(Some(TokenReviewIdentity { - namespace, - pod_name, - pod_uid, - })) -} - -fn parse_sa_username(username: &str) -> Option<(String, String)> { - let rest = username.strip_prefix("system:serviceaccount:")?; - let (namespace, sa_name) = rest.split_once(':')?; - if namespace.is_empty() || sa_name.is_empty() { - return None; - } - Some((namespace.to_string(), sa_name.to_string())) -} - -#[allow(clippy::result_large_err)] -fn user_extra_one(user: &UserInfo, key: &str) -> Result { - let Some(values) = user.extra.as_ref().and_then(|extra| extra.get(key)) else { - return Err(Status::permission_denied("SA token is not pod-bound")); - }; - if values.len() != 1 || values[0].is_empty() { - return Err(Status::permission_denied( - "SA token has invalid pod binding", - )); - } - Ok(values[0].clone()) -} - -#[allow(clippy::result_large_err)] -fn pod_sandbox_id(pod: &Pod) -> Result { - let sandbox_id = pod - .metadata - .annotations - .as_ref() - .and_then(|a| a.get(SANDBOX_ID_ANNOTATION)) - .cloned() - .unwrap_or_default(); - if sandbox_id.is_empty() { - return Err(Status::permission_denied( - "pod is not bound to a sandbox identity", - )); - } - Ok(sandbox_id) -} - -#[allow(clippy::result_large_err)] -fn sandbox_owner_reference(pod: &Pod) -> Result { - let owner_refs = pod.metadata.owner_references.as_deref().unwrap_or_default(); - let mut sandbox_refs = owner_refs - .iter() - .filter(|owner| is_supported_sandbox_owner_reference(owner)); - let Some(owner) = sandbox_refs.next() else { - let unsupported_sandbox_api_versions = owner_refs - .iter() - .filter(|owner| owner.kind == SANDBOX_KIND) - .map(|owner| owner.api_version.as_str()) - .collect::>(); - if !unsupported_sandbox_api_versions.is_empty() { - warn!( - api_versions = ?unsupported_sandbox_api_versions, - supported_api_versions = ?[ - SANDBOX_API_VERSION_FULL_V1BETA1, - SANDBOX_API_VERSION_FULL_V1ALPHA1, - ], - "pod Sandbox ownerReference uses unsupported apiVersion" - ); - } - return Err(Status::permission_denied( - "pod is not controlled by an OpenShell Sandbox", - )); - }; - if sandbox_refs.next().is_some() { - return Err(Status::permission_denied( - "pod has multiple OpenShell Sandbox owners", - )); - } - if owner.controller != Some(true) { - return Err(Status::permission_denied( - "pod Sandbox ownerReference is not controlling", - )); - } - if owner.name.is_empty() || owner.uid.is_empty() { - return Err(Status::permission_denied( - "pod Sandbox ownerReference is incomplete", - )); - } - Ok(SandboxOwnerReference { - api_version: owner.api_version.clone(), - name: owner.name.clone(), - uid: owner.uid.clone(), - }) -} - -fn is_supported_sandbox_owner_reference( - owner: &k8s_openapi::apimachinery::pkg::apis::meta::v1::OwnerReference, -) -> bool { - owner.kind == SANDBOX_KIND - && matches!( - owner.api_version.as_str(), - SANDBOX_API_VERSION_FULL_V1BETA1 | SANDBOX_API_VERSION_FULL_V1ALPHA1 - ) -} - -fn should_try_next_sandbox_api_version(err: &KubeError) -> bool { - // Kubernetes returns a structured 404 for some missing API resources and a - // raw "404 page not found" body for others. Both mean the probed - // group/version is unavailable and the next supported Sandbox API version - // should be tried. - matches!(err, KubeError::Api(api) if api.code == 404) -} - -#[allow(clippy::result_large_err)] -fn validate_sandbox_owner_reference( - owner: &SandboxOwnerReference, - sandbox_id: &str, - sandbox_cr: &DynamicObject, -) -> Result<(), Status> { - let actual_uid = sandbox_cr.metadata.uid.as_deref().unwrap_or_default(); - if actual_uid != owner.uid { - warn!( - sandbox_owner = %owner.name, - owner_uid = %owner.uid, - actual_uid = %actual_uid, - "pod Sandbox ownerReference UID does not match live Sandbox CR" - ); - return Err(Status::permission_denied("sandbox owner UID mismatch")); - } - - let actual_sandbox_id = sandbox_cr - .metadata - .labels - .as_ref() - .and_then(|labels| labels.get(SANDBOX_ID_LABEL)) - .map(String::as_str) - .unwrap_or_default(); - if actual_sandbox_id != sandbox_id { - warn!( - sandbox_owner = %owner.name, - owner_uid = %owner.uid, - pod_sandbox_id = %sandbox_id, - cr_sandbox_id = %actual_sandbox_id, - "pod sandbox annotation does not match owning Sandbox CR label" - ); - return Err(Status::permission_denied("sandbox owner ID mismatch")); - } - - Ok(()) -} - -#[cfg(test)] -pub mod test_support { - use super::*; - use std::sync::Mutex; - - /// Fake resolver for unit tests. Returns the configured outcome on - /// every call and records the tokens it observed. - pub struct FakeResolver { - pub outcome: Result, Status>, - pub seen_tokens: Mutex>, - } - - impl FakeResolver { - pub fn returning(outcome: Result, Status>) -> Self { - Self { - outcome, - seen_tokens: Mutex::new(Vec::new()), - } - } - } - - #[async_trait] - impl K8sIdentityResolver for FakeResolver { - async fn resolve(&self, token: &str) -> Result, Status> { - self.seen_tokens.lock().unwrap().push(token.to_string()); - match &self.outcome { - Ok(opt) => Ok(opt.clone()), - Err(s) => Err(Status::new(s.code(), s.message())), - } - } - } -} - -#[cfg(test)] -mod tests { - use super::test_support::FakeResolver; - use super::*; - use k8s_openapi::apimachinery::pkg::apis::meta::v1::OwnerReference; - use std::collections::BTreeMap; - - fn bearer_headers(token: &str) -> http::HeaderMap { - let mut h = http::HeaderMap::new(); - h.insert( - "authorization", - http::HeaderValue::from_str(&format!("Bearer {token}")).unwrap(), - ); - h - } - - fn kube_api_error(code: u16, message: &str) -> KubeError { - KubeError::Api(kube::core::ErrorResponse { - status: if code == 404 { - "404 Not Found".to_string() - } else { - "Failure".to_string() - }, - message: message.to_string(), - reason: "Failed to parse error data".to_string(), - code, - }) - } - - #[test] - fn sandbox_api_version_probe_retries_on_structured_and_raw_404() { - let structured = kube_api_error(404, "could not find the requested resource"); - assert!(should_try_next_sandbox_api_version(&structured)); - - let raw = kube_api_error(404, "404 page not found\n"); - assert!(should_try_next_sandbox_api_version(&raw)); - } - - #[test] - fn sandbox_api_version_probe_keeps_non_404_errors() { - let err = kube_api_error(403, "sandboxes.agents.x-k8s.io is forbidden"); - assert!(!should_try_next_sandbox_api_version(&err)); - } - - fn token_review_status( - authenticated: bool, - audiences: Vec<&str>, - username: &str, - extra: Vec<(&str, &str)>, - ) -> TokenReviewStatus { - TokenReviewStatus { - authenticated: Some(authenticated), - audiences: Some(audiences.into_iter().map(str::to_string).collect()), - error: None, - user: Some(UserInfo { - username: Some(username.to_string()), - uid: Some("sa-uid".to_string()), - groups: Some(vec![ - "system:serviceaccounts".to_string(), - "system:serviceaccounts:openshell".to_string(), - "system:authenticated".to_string(), - ]), - extra: Some( - extra - .into_iter() - .map(|(k, v)| (k.to_string(), vec![v.to_string()])) - .collect::>(), - ), - }), - } - } - - fn sandbox_owner(name: &str, uid: &str) -> OwnerReference { - sandbox_owner_with_api_version(SANDBOX_API_VERSION_FULL_V1BETA1, name, uid) - } - - fn sandbox_owner_with_api_version(api_version: &str, name: &str, uid: &str) -> OwnerReference { - OwnerReference { - api_version: api_version.to_string(), - block_owner_deletion: None, - controller: Some(true), - kind: SANDBOX_KIND.to_string(), - name: name.to_string(), - uid: uid.to_string(), - } - } - - fn pod_with_owner_refs(owner_references: Vec) -> Pod { - Pod { - metadata: ObjectMeta { - owner_references: Some(owner_references), - ..Default::default() - }, - ..Default::default() - } - } - - fn pod_with_sandbox_id(sandbox_id: Option<&str>) -> Pod { - Pod { - metadata: ObjectMeta { - annotations: sandbox_id.map(|id| { - BTreeMap::from([(SANDBOX_ID_ANNOTATION.to_string(), id.to_string())]) - }), - ..Default::default() - }, - ..Default::default() - } - } - - fn sandbox_cr(name: &str, uid: &str, sandbox_id: &str) -> DynamicObject { - let sandbox_gvk = - GroupVersionKind::gvk(SANDBOX_API_GROUP, SANDBOX_API_VERSION_V1BETA1, SANDBOX_KIND); - let sandbox_resource = ApiResource::from_gvk(&sandbox_gvk); - let mut cr = DynamicObject::new(name, &sandbox_resource); - cr.metadata.uid = Some(uid.to_string()); - cr.metadata.labels = Some(BTreeMap::from([( - SANDBOX_ID_LABEL.to_string(), - sandbox_id.to_string(), - )])); - cr - } - - fn exact_validator(ns: &str) -> NamespaceValidator { - NamespaceValidator::Exact(ns.to_string()) - } - - #[test] - fn token_review_identity_extracts_pod_binding() { - let status = token_review_status( - true, - vec!["openshell-gateway"], - "system:serviceaccount:openshell:default", - vec![ - (POD_NAME_EXTRA, "openshell-sandbox-a"), - (POD_UID_EXTRA, "uid-a"), - ], - ); - - let validator = exact_validator("openshell"); - let identity = token_review_identity(&status, "openshell-gateway", &validator, "default") - .unwrap() - .expect("authenticated token should resolve"); - - assert_eq!(identity.namespace, "openshell"); - assert_eq!(identity.pod_name, "openshell-sandbox-a"); - assert_eq!(identity.pod_uid, "uid-a"); - } - - #[test] - fn token_review_identity_returns_none_when_not_authenticated() { - let status = TokenReviewStatus { - authenticated: Some(false), - error: Some("invalid audience".to_string()), - ..Default::default() - }; - let validator = exact_validator("openshell"); - - assert!( - token_review_identity(&status, "openshell-gateway", &validator, "default") - .unwrap() - .is_none() - ); - } - - #[test] - fn token_review_identity_requires_expected_audience() { - let status = token_review_status( - true, - vec!["kubernetes.default.svc"], - "system:serviceaccount:openshell:default", - vec![ - (POD_NAME_EXTRA, "openshell-sandbox-a"), - (POD_UID_EXTRA, "uid-a"), - ], - ); - let validator = exact_validator("openshell"); - - let err = token_review_identity(&status, "openshell-gateway", &validator, "default") - .expect_err("wrong audience must fail closed"); - assert_eq!(err.code(), tonic::Code::Unauthenticated); - } - - #[test] - fn token_review_identity_requires_sandbox_namespace() { - let status = token_review_status( - true, - vec!["openshell-gateway"], - "system:serviceaccount:other:default", - vec![ - (POD_NAME_EXTRA, "openshell-sandbox-a"), - (POD_UID_EXTRA, "uid-a"), - ], - ); - let validator = exact_validator("openshell"); - - let err = token_review_identity(&status, "openshell-gateway", &validator, "default") - .expect_err("other namespace must be rejected"); - assert_eq!(err.code(), tonic::Code::PermissionDenied); - } - - #[test] - fn token_review_identity_requires_configured_service_account() { - let status = token_review_status( - true, - vec!["openshell-gateway"], - "system:serviceaccount:openshell:other", - vec![ - (POD_NAME_EXTRA, "openshell-sandbox-a"), - (POD_UID_EXTRA, "uid-a"), - ], - ); - let validator = exact_validator("openshell"); - - let err = token_review_identity(&status, "openshell-gateway", &validator, "default") - .expect_err("other service account must be rejected"); - assert_eq!(err.code(), tonic::Code::PermissionDenied); - } - - #[test] - fn token_review_identity_requires_pod_bound_extras() { - let status = token_review_status( - true, - vec!["openshell-gateway"], - "system:serviceaccount:openshell:default", - vec![], - ); - let validator = exact_validator("openshell"); - - let err = token_review_identity(&status, "openshell-gateway", &validator, "default") - .expect_err("non pod-bound tokens must be rejected"); - assert_eq!(err.code(), tonic::Code::PermissionDenied); - } - - #[test] - fn namespace_validator_exact_accepts_matching() { - let v = NamespaceValidator::Exact("openshell".to_string()); - assert!(v.accepts("openshell")); - assert!(!v.accepts("other")); - } - - #[test] - fn namespace_validator_prefix_accepts_managed_namespaces() { - let v = NamespaceValidator::Prefix("openshell-gw1-".to_string()); - assert!(v.accepts("openshell-gw1-workspace-a")); - assert!(v.accepts("openshell-gw1-default")); - assert!(!v.accepts("openshell-gw2-workspace-a")); - assert!(!v.accepts("other")); - } - - #[test] - fn namespace_validator_allowlist_accepts_known_namespaces() { - let al = OperatorNamespaceAllowlist::from_set(std::collections::BTreeSet::from([ - "ns-a".to_string(), - "ns-b".to_string(), - ])); - let v = NamespaceValidator::Allowlist(al); - assert!(v.accepts("ns-a")); - assert!(v.accepts("ns-b")); - assert!(!v.accepts("ns-c")); - } - - #[test] - fn token_review_identity_prefix_validator_accepts_managed_namespace() { - let status = token_review_status( - true, - vec!["openshell-gateway"], - "system:serviceaccount:openshell-gw1-workspace-a:default", - vec![ - (POD_NAME_EXTRA, "openshell-sandbox-a"), - (POD_UID_EXTRA, "uid-a"), - ], - ); - let validator = NamespaceValidator::Prefix("openshell-gw1-".to_string()); - - let identity = token_review_identity(&status, "openshell-gateway", &validator, "default") - .unwrap() - .expect("managed namespace token should resolve"); - assert_eq!(identity.namespace, "openshell-gw1-workspace-a"); - } - - #[test] - fn parse_sa_username_extracts_namespace_and_sa() { - let (ns, sa) = parse_sa_username("system:serviceaccount:openshell:default").unwrap(); - assert_eq!(ns, "openshell"); - assert_eq!(sa, "default"); - - assert!(parse_sa_username("system:node:nodename").is_none()); - assert!(parse_sa_username("system:serviceaccount::default").is_none()); - assert!(parse_sa_username("system:serviceaccount:ns:").is_none()); - } - - #[test] - fn pod_sandbox_id_requires_annotation() { - assert_eq!( - pod_sandbox_id(&pod_with_sandbox_id(Some("sandbox-id-a"))).unwrap(), - "sandbox-id-a" - ); - - let err = pod_sandbox_id(&pod_with_sandbox_id(None)) - .expect_err("missing sandbox-id annotation must fail"); - assert_eq!(err.code(), tonic::Code::PermissionDenied); - } - - #[test] - fn sandbox_owner_reference_extracts_controlling_sandbox_owner() { - let pod = pod_with_owner_refs(vec![sandbox_owner("sandbox-a", "cr-uid-a")]); - - let owner = sandbox_owner_reference(&pod).expect("expected Sandbox owner"); - - assert_eq!( - owner, - SandboxOwnerReference { - api_version: SANDBOX_API_VERSION_FULL_V1BETA1.to_string(), - name: "sandbox-a".to_string(), - uid: "cr-uid-a".to_string(), - } - ); - } - - #[test] - fn sandbox_owner_reference_accepts_v1alpha1_owner() { - let pod = pod_with_owner_refs(vec![sandbox_owner_with_api_version( - SANDBOX_API_VERSION_FULL_V1ALPHA1, - "sandbox-a", - "cr-uid-a", - )]); - - let owner = sandbox_owner_reference(&pod).expect("expected v1alpha1 Sandbox owner"); - - assert_eq!( - owner, - SandboxOwnerReference { - api_version: SANDBOX_API_VERSION_FULL_V1ALPHA1.to_string(), - name: "sandbox-a".to_string(), - uid: "cr-uid-a".to_string(), - } - ); - } - - #[test] - fn sandbox_owner_reference_rejects_missing_owner() { - let pod = pod_with_owner_refs(vec![]); - - let err = sandbox_owner_reference(&pod).expect_err("missing owner must fail"); - - assert_eq!(err.code(), tonic::Code::PermissionDenied); - } - - #[test] - fn sandbox_owner_reference_rejects_unsupported_sandbox_api_version() { - let pod = pod_with_owner_refs(vec![sandbox_owner_with_api_version( - "agents.x-k8s.io/v1", - "sandbox-a", - "cr-uid-a", - )]); - - let err = - sandbox_owner_reference(&pod).expect_err("unsupported apiVersion must fail closed"); - - assert_eq!(err.code(), tonic::Code::PermissionDenied); - } - - #[test] - fn sandbox_owner_reference_requires_controlling_owner() { - let mut owner = sandbox_owner("sandbox-a", "cr-uid-a"); - owner.controller = Some(false); - let pod = pod_with_owner_refs(vec![owner]); - - let err = sandbox_owner_reference(&pod).expect_err("non-controller owner must fail"); - - assert_eq!(err.code(), tonic::Code::PermissionDenied); - } - - #[test] - fn sandbox_owner_reference_rejects_ambiguous_sandbox_owners() { - let pod = pod_with_owner_refs(vec![ - sandbox_owner("sandbox-a", "cr-uid-a"), - sandbox_owner("sandbox-b", "cr-uid-b"), - ]); - - let err = sandbox_owner_reference(&pod).expect_err("multiple owners must fail"); - - assert_eq!(err.code(), tonic::Code::PermissionDenied); - } - - #[test] - fn validate_sandbox_owner_reference_requires_matching_cr_uid_and_label() { - let owner = SandboxOwnerReference { - api_version: SANDBOX_API_VERSION_FULL_V1BETA1.to_string(), - name: "sandbox-a".to_string(), - uid: "cr-uid-a".to_string(), - }; - let cr = sandbox_cr("sandbox-a", "cr-uid-a", "sandbox-id-a"); - validate_sandbox_owner_reference(&owner, "sandbox-id-a", &cr) - .expect("matching CR should be accepted"); - - let wrong_uid = sandbox_cr("sandbox-a", "cr-uid-b", "sandbox-id-a"); - let err = validate_sandbox_owner_reference(&owner, "sandbox-id-a", &wrong_uid) - .expect_err("wrong CR UID must fail"); - assert_eq!(err.code(), tonic::Code::PermissionDenied); - - let wrong_label = sandbox_cr("sandbox-a", "cr-uid-a", "sandbox-id-b"); - let err = validate_sandbox_owner_reference(&owner, "sandbox-id-a", &wrong_label) - .expect_err("wrong sandbox-id label must fail"); - assert_eq!(err.code(), tonic::Code::PermissionDenied); - } - - #[tokio::test] - async fn authenticates_on_issue_path_only() { - let resolved = ResolvedK8sIdentity { - sandbox_id: "sandbox-a".to_string(), - pod_name: "openshell-sandbox-a".to_string(), - pod_uid: "uid-a".to_string(), - }; - let fake = Arc::new(FakeResolver::returning(Ok(Some(resolved)))); - let auth = K8sServiceAccountAuthenticator::new(fake.clone()); - - let on_issue = auth - .authenticate(&bearer_headers("sa-jwt"), ISSUE_SANDBOX_TOKEN_PATH) - .await - .unwrap() - .expect("expected principal"); - match on_issue { - Principal::Sandbox(p) => { - assert_eq!(p.sandbox_id, "sandbox-a"); - assert!(matches!( - p.source, - SandboxIdentitySource::K8sServiceAccount { .. } - )); - } - _ => panic!("expected sandbox principal"), - } - - let off_issue = auth - .authenticate( - &bearer_headers("sa-jwt"), - "/openshell.v1.OpenShell/GetSandboxConfig", - ) - .await - .unwrap(); - assert!( - off_issue.is_none(), - "K8s SA authenticator must be scoped to IssueSandboxToken" - ); - assert_eq!( - fake.seen_tokens.lock().unwrap().len(), - 1, - "off-path call must not consult the apiserver" - ); - } - - #[tokio::test] - async fn missing_bearer_yields_none() { - let fake = Arc::new(FakeResolver::returning(Ok(None))); - let auth = K8sServiceAccountAuthenticator::new(fake); - let result = auth - .authenticate(&http::HeaderMap::new(), ISSUE_SANDBOX_TOKEN_PATH) - .await - .unwrap(); - assert!(result.is_none()); - } - - #[tokio::test] - async fn resolver_returning_none_falls_through() { - let fake = Arc::new(FakeResolver::returning(Ok(None))); - let auth = K8sServiceAccountAuthenticator::new(fake); - let result = auth - .authenticate( - &bearer_headers("not-a-real-sa-token"), - ISSUE_SANDBOX_TOKEN_PATH, - ) - .await - .unwrap(); - assert!(result.is_none(), "non-authenticating tokens fall through"); - } - - #[tokio::test] - async fn pod_without_annotation_is_rejected() { - let resolved = ResolvedK8sIdentity { - sandbox_id: String::new(), - pod_name: "stray-pod".to_string(), - pod_uid: "uid".to_string(), - }; - let fake = Arc::new(FakeResolver::returning(Ok(Some(resolved)))); - let auth = K8sServiceAccountAuthenticator::new(fake); - let err = auth - .authenticate(&bearer_headers("sa-jwt"), ISSUE_SANDBOX_TOKEN_PATH) - .await - .expect_err("unbound pod must be rejected"); - assert_eq!(err.code(), tonic::Code::PermissionDenied); - } - - #[tokio::test] - async fn resolver_error_propagates() { - let fake = Arc::new(FakeResolver::returning(Err(Status::unavailable( - "apiserver down", - )))); - let auth = K8sServiceAccountAuthenticator::new(fake); - let err = auth - .authenticate(&bearer_headers("sa-jwt"), ISSUE_SANDBOX_TOKEN_PATH) - .await - .expect_err("resolver error must propagate"); - assert_eq!(err.code(), tonic::Code::Unavailable); - } -} diff --git a/crates/openshell-server/src/auth/mod.rs b/crates/openshell-server/src/auth/mod.rs index bedbebe015..b39ff7bfa9 100644 --- a/crates/openshell-server/src/auth/mod.rs +++ b/crates/openshell-server/src/auth/mod.rs @@ -10,12 +10,12 @@ pub mod authenticator; pub mod authz; +pub mod compute_driver; pub mod descriptor_authz; pub mod extension_mint_limit; pub mod guard; mod http; pub mod identity; -pub mod k8s_sa; pub mod method_authz; pub mod oidc; pub mod principal; diff --git a/crates/openshell-server/src/auth/principal.rs b/crates/openshell-server/src/auth/principal.rs index 1d4cb7276c..9567cc62d2 100644 --- a/crates/openshell-server/src/auth/principal.rs +++ b/crates/openshell-server/src/auth/principal.rs @@ -70,7 +70,8 @@ pub enum SandboxIdentitySource { /// Per-sandbox client certificate. Reserved for channel-bound sandbox /// identity. BootstrapCert { fingerprint: String }, - /// K8s `ServiceAccount` token used to bootstrap a gateway-minted JWT - /// via `IssueSandboxToken`. Populated only on that one RPC path. - K8sServiceAccount { pod_name: String, pod_uid: String }, + /// Driver-native credential used to bootstrap a gateway-minted JWT via + /// `IssueSandboxToken`. The named compute driver authenticated only the + /// sandbox identity; the gateway still authorizes the exchange. + ComputeDriver { driver_name: String }, } diff --git a/crates/openshell-server/src/compute/driver_config.rs b/crates/openshell-server/src/compute/driver_config.rs index fad71efd36..d4237d24d7 100644 --- a/crates/openshell-server/src/compute/driver_config.rs +++ b/crates/openshell-server/src/compute/driver_config.rs @@ -75,51 +75,6 @@ pub struct RemoteDriverConfig { pub socket_path: PathBuf, } -#[derive(Debug, Clone, Deserialize)] -#[serde(default)] -pub struct KubernetesSaBootstrapConfig { - pub namespace: String, - pub service_account_name: String, - pub workspace_mode: String, - pub gateway_id: String, -} - -impl Default for KubernetesSaBootstrapConfig { - fn default() -> Self { - Self { - namespace: "openshell".to_string(), - service_account_name: "default".to_string(), - workspace_mode: "shared".to_string(), - gateway_id: "openshell".to_string(), - } - } -} - -pub fn kubernetes_sa_bootstrap_config( - file: Option<&config_file::ConfigFile>, -) -> Result { - let Some(file) = file else { - return Err(Error::config( - "K8s ServiceAccount bootstrap requires [openshell.drivers.kubernetes] when sandbox JWT issuing is enabled in-cluster", - )); - }; - if !file.openshell.drivers.contains_key("kubernetes") { - return Err(Error::config( - "K8s ServiceAccount bootstrap requires [openshell.drivers.kubernetes] when sandbox JWT issuing is enabled in-cluster", - )); - } - let merged = config_file::driver_table( - "kubernetes", - &file.openshell.gateway, - file.openshell.drivers.get("kubernetes"), - ); - merged.try_into().map_err(|error| { - Error::config(format!( - "invalid Kubernetes ServiceAccount bootstrap config: {error}" - )) - }) -} - pub fn driver_config_from_context( context: DriverStartupContext<'_>, driver_name: &str, @@ -234,29 +189,6 @@ service_account_name = "sandbox-sa" ); } - #[test] - fn kubernetes_sa_bootstrap_uses_public_gateway_config() { - let file: config_file::ConfigFile = toml::from_str( - r#" -[openshell.gateway] -sandbox_namespace = "sandboxes" - -[openshell.drivers.kubernetes] -socket_path = "/run/openshell/kubernetes.sock" -workspace_mode = "managed" -gateway_id = "gateway-a" -service_account_name = "sandbox-sa" -"#, - ) - .expect("valid config"); - - let cfg = kubernetes_sa_bootstrap_config(Some(&file)).expect("bootstrap config"); - assert_eq!(cfg.namespace, "sandboxes"); - assert_eq!(cfg.workspace_mode, "managed"); - assert_eq!(cfg.gateway_id, "gateway-a"); - assert_eq!(cfg.service_account_name, "sandbox-sa"); - } - #[test] fn remote_driver_config_uses_endpoint_override_without_file() { let endpoint_overrides = diff --git a/crates/openshell-server/src/compute/mod.rs b/crates/openshell-server/src/compute/mod.rs index 65411d28f1..ec6a96e310 100644 --- a/crates/openshell-server/src/compute/mod.rs +++ b/crates/openshell-server/src/compute/mod.rs @@ -32,17 +32,18 @@ use futures::{Stream, StreamExt}; use hyper_util::rt::TokioIo; use openshell_core::ComputeDriverKind; use openshell_core::proto::compute::v1::{ - CreateSandboxRequest, DeleteSandboxRequest, DeleteWorkspaceRequest, DeleteWorkspaceResponse, - DriverCondition, DriverPlatformEvent, DriverResourceRequirements, DriverSandbox, - DriverSandboxSpec, DriverSandboxStatus, DriverSandboxTemplate, EnsureWorkspaceRequest, - EnsureWorkspaceResponse, GatewayListenerRequirement as ProtoGatewayListenerRequirement, - GetCapabilitiesRequest, GetGatewayListenerRequirementsRequest, - GetGatewayListenerRequirementsResponse, GetSandboxRequest, - GpuResourceRequirements as DriverGpuResourceRequirements, ListSandboxesRequest, - ResourceRequirements as DriverSandboxResourceRequirements, StartSandboxRequest, - StopSandboxRequest, ValidateSandboxCreateRequest, WatchSandboxesEvent, WatchSandboxesRequest, - compute_driver_client::ComputeDriverClient, compute_driver_server::ComputeDriver, - gateway_listener_requirement::Selector, watch_sandboxes_event, + AuthenticateSandboxRequest, CreateSandboxRequest, DeleteSandboxRequest, DeleteWorkspaceRequest, + DeleteWorkspaceResponse, DriverCondition, DriverPlatformEvent, DriverResourceRequirements, + DriverSandbox, DriverSandboxSpec, DriverSandboxStatus, DriverSandboxTemplate, + EnsureWorkspaceRequest, EnsureWorkspaceResponse, + GatewayListenerRequirement as ProtoGatewayListenerRequirement, GetCapabilitiesRequest, + GetGatewayListenerRequirementsRequest, GetGatewayListenerRequirementsResponse, + GetSandboxRequest, GpuResourceRequirements as DriverGpuResourceRequirements, + ListSandboxesRequest, ResourceRequirements as DriverSandboxResourceRequirements, + StartSandboxRequest, StopSandboxRequest, ValidateSandboxCreateRequest, WatchSandboxesEvent, + WatchSandboxesRequest, compute_driver_client::ComputeDriverClient, + compute_driver_server::ComputeDriver, gateway_listener_requirement::Selector, + watch_sandboxes_event, }; use openshell_core::proto::{ PlatformEvent, Sandbox, SandboxCondition, SandboxPhase, SandboxSpec, SandboxStatus, @@ -280,6 +281,8 @@ pub struct ComputeDriverInfoSnapshot { pub driver_version: String, /// Whether the driver asks the gateway to reconcile compute across restarts. pub gateway_manages_lifecycle: bool, + /// Whether the driver authenticates driver-native sandbox credentials. + pub supports_sandbox_authentication: bool, } /// Interval between store-vs-backend reconciliation sweeps. @@ -453,6 +456,17 @@ impl ComputeDriver for RemoteComputeDriver { client.get_capabilities(request).await } + async fn authenticate_sandbox( + &self, + request: Request, + ) -> Result< + tonic::Response, + Status, + > { + let mut client = self.client(); + client.authenticate_sandbox(request).await + } + async fn get_gateway_listener_requirements( &self, request: Request, @@ -617,6 +631,7 @@ impl ComputeRuntime { driver_name: capabilities.driver_name, driver_version: capabilities.driver_version, gateway_manages_lifecycle: capabilities.gateway_manages_lifecycle, + supports_sandbox_authentication: capabilities.supports_sandbox_authentication, }; let default_image = capabilities.default_image; let gateway_listener_requirements = match driver @@ -833,6 +848,33 @@ impl ComputeRuntime { self.driver_info.name.parse().ok() } + #[must_use] + pub fn selected_driver_name(&self) -> &str { + &self.driver_info.name + } + + #[must_use] + pub fn supports_sandbox_authentication(&self) -> bool { + self.driver_info.supports_sandbox_authentication + } + + pub(crate) async fn authenticate_sandbox(&self, credential: &str) -> Result { + if !self.supports_sandbox_authentication() { + return Err(Status::unimplemented( + "selected compute driver does not authenticate sandbox credentials", + )); + } + let request = AuthenticateSandboxRequest { + credential: credential.to_string(), + }; + self.driver + .call("driver.authenticate_sandbox", None, |driver| async move { + driver.authenticate_sandbox(Request::new(request)).await + }) + .await + .map(|response| response.into_inner().sandbox_id) + } + #[must_use] pub(crate) fn gateway_listener_requirements(&self) -> &[GatewayListenerRequirement] { &self.gateway_listener_requirements @@ -4075,9 +4117,10 @@ fn is_terminal_failure_reason(reason: &str) -> bool { } #[cfg(test)] -#[derive(Debug, Default)] +#[derive(Debug)] pub struct NoopTestDriver { workspace_delete_failures: std::sync::atomic::AtomicUsize, + sandbox_authentication: Option>, } #[cfg(test)] @@ -4085,6 +4128,31 @@ impl NoopTestDriver { pub fn failing_workspace_deletes(count: usize) -> Self { Self { workspace_delete_failures: std::sync::atomic::AtomicUsize::new(count), + sandbox_authentication: None, + } + } + + pub fn authenticating_sandbox(sandbox_id: impl Into) -> Self { + Self { + workspace_delete_failures: std::sync::atomic::AtomicUsize::new(0), + sandbox_authentication: Some(Ok(sandbox_id.into())), + } + } + + pub fn failing_sandbox_authentication(code: Code, message: impl Into) -> Self { + Self { + workspace_delete_failures: std::sync::atomic::AtomicUsize::new(0), + sandbox_authentication: Some(Err((code, message.into()))), + } + } +} + +#[cfg(test)] +impl Default for NoopTestDriver { + fn default() -> Self { + Self { + workspace_delete_failures: std::sync::atomic::AtomicUsize::new(0), + sandbox_authentication: None, } } } @@ -4092,6 +4160,26 @@ impl NoopTestDriver { #[cfg(test)] #[tonic::async_trait] impl ComputeDriver for NoopTestDriver { + async fn authenticate_sandbox( + &self, + _request: Request, + ) -> Result< + tonic::Response, + Status, + > { + match &self.sandbox_authentication { + Some(Ok(sandbox_id)) => Ok(tonic::Response::new( + openshell_core::proto::compute::v1::AuthenticateSandboxResponse { + sandbox_id: sandbox_id.clone(), + }, + )), + Some(Err((code, message))) => Err(Status::new(*code, message.clone())), + None => Err(Status::unimplemented( + "test driver does not authenticate sandbox credentials", + )), + } + } + type WatchSandboxesStream = DriverWatchStream; async fn get_capabilities( @@ -4105,6 +4193,7 @@ impl ComputeDriver for NoopTestDriver { driver_version: "test".to_string(), default_image: "openshell/sandbox:test".to_string(), gateway_manages_lifecycle: false, + supports_sandbox_authentication: self.sandbox_authentication.is_some(), }, )) } @@ -4239,6 +4328,7 @@ pub async fn new_test_runtime_with_driver( driver_name: &str, driver: Arc, ) -> ComputeRuntime { + let supports_sandbox_authentication = driver.sandbox_authentication.is_some(); ComputeRuntime { driver: TracedDriver::new(driver, "test".to_string()), driver_info: ComputeDriverInfoSnapshot { @@ -4246,6 +4336,7 @@ pub async fn new_test_runtime_with_driver( driver_name: driver_name.to_string(), driver_version: "test".to_string(), gateway_manages_lifecycle: false, + supports_sandbox_authentication, }, driver_process: None, default_image: "openshell/sandbox:test".to_string(), @@ -4398,6 +4489,18 @@ mod tests { #[tonic::async_trait] impl ComputeDriver for TestDriver { + async fn authenticate_sandbox( + &self, + _request: Request, + ) -> Result< + tonic::Response, + Status, + > { + Err(Status::unimplemented( + "test driver does not authenticate sandbox credentials", + )) + } + type WatchSandboxesStream = DriverWatchStream; async fn get_capabilities( @@ -4409,6 +4512,7 @@ mod tests { driver_version: "test".to_string(), default_image: "openshell/sandbox:test".to_string(), gateway_manages_lifecycle: false, + supports_sandbox_authentication: false, })) } @@ -4716,6 +4820,18 @@ mod tests { #[tonic::async_trait] impl ComputeDriver for ControlledDriver { + async fn authenticate_sandbox( + &self, + _request: Request, + ) -> Result< + tonic::Response, + Status, + > { + Err(Status::unimplemented( + "test driver does not authenticate sandbox credentials", + )) + } + type WatchSandboxesStream = DriverWatchStream; async fn get_capabilities( @@ -4727,6 +4843,7 @@ mod tests { driver_version: "test".to_string(), default_image: "openshell/sandbox:test".to_string(), gateway_manages_lifecycle: false, + supports_sandbox_authentication: false, })) } @@ -4930,6 +5047,7 @@ mod tests { driver_name: driver_name.to_string(), driver_version: "test".to_string(), gateway_manages_lifecycle: false, + supports_sandbox_authentication: false, }, driver_process: None, default_image: "openshell/sandbox:test".to_string(), @@ -5840,6 +5958,18 @@ mod tests { #[tonic::async_trait] impl ComputeDriver for FailingDriver { + async fn authenticate_sandbox( + &self, + _request: Request, + ) -> Result< + tonic::Response, + Status, + > { + Err(Status::unimplemented( + "test driver does not authenticate sandbox credentials", + )) + } + type WatchSandboxesStream = DriverWatchStream; async fn create_sandbox( diff --git a/crates/openshell-server/src/grpc/auth_rpc.rs b/crates/openshell-server/src/grpc/auth_rpc.rs index dca563a715..104d639584 100644 --- a/crates/openshell-server/src/grpc/auth_rpc.rs +++ b/crates/openshell-server/src/grpc/auth_rpc.rs @@ -5,7 +5,7 @@ //! //! Hosts authenticated identity RPCs: //! - `GetCurrentUser` — report the gateway-validated caller identity -//! - `IssueSandboxToken` — bootstrap exchange (K8s SA token → gateway JWT) +//! - `IssueSandboxToken` — driver-native bootstrap exchange → gateway JWT //! - `RefreshSandboxToken` — renew a still-valid gateway JWT //! //! Both end in a fresh gateway-signed JWT minted by @@ -71,13 +71,9 @@ pub async fn handle_issue_sandbox_token( )); }; - // Only the bootstrap K8s ServiceAccount path can mint a fresh gateway JWT - // via this RPC. Sandboxes already holding a gateway JWT use - // `RefreshSandboxToken` instead. - if !matches!( - sandbox.source, - SandboxIdentitySource::K8sServiceAccount { .. } - ) { + // Only a selected compute driver may establish the bootstrap sandbox + // identity. Sandboxes already holding a gateway JWT use refresh instead. + if !matches!(sandbox.source, SandboxIdentitySource::ComputeDriver { .. }) { debug!( sandbox_id = %sandbox.sandbox_id, "IssueSandboxToken rejected: non-bootstrap principal source" @@ -518,9 +514,8 @@ mod tests { req.extensions_mut() .insert(Principal::Sandbox(SandboxPrincipal { sandbox_id: "sandbox-a".to_string(), - source: SandboxIdentitySource::K8sServiceAccount { - pod_name: "pod-a".to_string(), - pod_uid: "uid-a".to_string(), + source: SandboxIdentitySource::ComputeDriver { + driver_name: "kubernetes".to_string(), }, trust_domain: Some("openshell".to_string()), })); @@ -541,9 +536,8 @@ mod tests { req.extensions_mut() .insert(Principal::Sandbox(SandboxPrincipal { sandbox_id: "sandbox-deleted".to_string(), - source: SandboxIdentitySource::K8sServiceAccount { - pod_name: "pod-a".to_string(), - pod_uid: "uid-a".to_string(), + source: SandboxIdentitySource::ComputeDriver { + driver_name: "kubernetes".to_string(), }, trust_domain: Some("openshell".to_string()), })); @@ -576,8 +570,8 @@ mod tests { } #[tokio::test] - async fn refresh_rejects_k8s_sa_principal() { - // K8s SA-bootstrap principals must use IssueSandboxToken, not + async fn refresh_rejects_compute_driver_principal() { + // Driver-bootstrap principals must use IssueSandboxToken, not // RefreshSandboxToken — the refresh path assumes a still-valid // gateway-minted JWT exists. use crate::auth::principal::SandboxIdentitySource; @@ -588,9 +582,8 @@ mod tests { req.extensions_mut() .insert(Principal::Sandbox(SandboxPrincipal { sandbox_id: "sandbox-a".to_string(), - source: SandboxIdentitySource::K8sServiceAccount { - pod_name: "pod-a".to_string(), - pod_uid: "uid-a".to_string(), + source: SandboxIdentitySource::ComputeDriver { + driver_name: "kubernetes".to_string(), }, trust_domain: Some("openshell".to_string()), })); diff --git a/crates/openshell-server/src/lib.rs b/crates/openshell-server/src/lib.rs index 2667611bcc..7dc91eccfa 100644 --- a/crates/openshell-server/src/lib.rs +++ b/crates/openshell-server/src/lib.rs @@ -312,10 +312,9 @@ pub struct ServerState { /// presenting a freshly minted token are recognized. pub sandbox_jwt_authenticator: Option>, - /// Optional K8s `ServiceAccount` authenticator that backs the - /// `IssueSandboxToken` bootstrap path. Only present when the gateway - /// runs in-cluster. - pub k8s_sa_authenticator: Option>, + /// Optional selected-driver authenticator for the `IssueSandboxToken` + /// bootstrap path. + pub compute_driver_authenticator: Option>, /// Gateway-wide gRPC request rate limiter shared by every multiplex path. pub(crate) grpc_rate_limiter: Option, @@ -416,7 +415,7 @@ impl ServerState { oidc_cache, sandbox_jwt_issuer: None, sandbox_jwt_authenticator: None, - k8s_sa_authenticator: None, + compute_driver_authenticator: None, grpc_rate_limiter, gateway_interceptors: None, provider_profile_sources: @@ -590,7 +589,7 @@ pub(crate) async fn run_server( gateway_tls_enabled: config.tls.is_some(), endpoint_overrides: &config.compute_driver_endpoints, }; - let (compute, operator_allowlist) = build_compute_runtime( + let (compute, _operator_allowlist) = build_compute_runtime( &config, driver_startup, compute_driver, @@ -660,43 +659,14 @@ pub(crate) async fn run_server( spawn_gateway_extension_token_refresh(issuer, gateway_extension_credentials); } - // K8s ServiceAccount bootstrap authenticator. Only constructed when - // the gateway is running in-cluster (kubelet provides the API host - // env var) and has a sandbox JWT issuer to mint replacements against; - // outside the cluster we can't call the apiserver's TokenReview API, - // and without the issuer there's nothing to exchange the SA token for. - #[cfg(not(target_os = "windows"))] - if state.sandbox_jwt_issuer.is_some() && std::env::var_os("KUBERNETES_SERVICE_HOST").is_some() { - // Pod lookups and TokenReview identity checks must match the sandbox - // namespace and service account used by the Kubernetes driver. - let kubernetes_config = - compute::driver_config::kubernetes_sa_bootstrap_config(config_file.as_ref())?; - let sandbox_namespace = kubernetes_config.namespace.clone(); - let sandbox_service_account = kubernetes_config.service_account_name.clone(); - let namespace_validator = - kubernetes_namespace_validator(&kubernetes_config, &operator_allowlist)?; - match kube::Client::try_default().await { - Ok(client) => { - let resolver = Arc::new(auth::k8s_sa::LiveK8sResolver::new( - client, - namespace_validator, - "openshell-gateway".to_string(), - sandbox_service_account.clone(), - )); - let authenticator = auth::k8s_sa::K8sServiceAccountAuthenticator::new(resolver); - state.k8s_sa_authenticator = Some(Arc::new(authenticator)); - info!( - namespace = %sandbox_namespace, - service_account = %sandbox_service_account, - "K8s ServiceAccount bootstrap authenticator enabled" - ); - } - Err(e) => warn!( - error = %e, - "in-cluster K8s client construction failed; \ - K8s ServiceAccount bootstrap is disabled" - ), - } + if state.sandbox_jwt_issuer.is_some() && state.compute.supports_sandbox_authentication() { + state.compute_driver_authenticator = Some(Arc::new( + auth::compute_driver::ComputeDriverAuthenticator::new(state.compute.clone()), + )); + info!( + driver = state.compute.selected_driver_name(), + "compute-driver sandbox bootstrap authenticator enabled" + ); } let state = Arc::new(state); @@ -1067,51 +1037,6 @@ fn unsupported_builtin_compute_driver(driver: ComputeDriverKind) -> compute::Com type OperatorAllowlistArc = Option; pub use compute::{DriverWatchStream, SharedComputeDriver}; -fn kubernetes_namespace_validator( - config: &compute::driver_config::KubernetesSaBootstrapConfig, - operator_allowlist: &OperatorAllowlistArc, -) -> Result { - match config.workspace_mode.as_str() { - "shared" => Ok(auth::k8s_sa::NamespaceValidator::Exact( - config.namespace.clone(), - )), - "managed" => Ok(auth::k8s_sa::NamespaceValidator::Prefix(format!( - "openshell-{}-", - config.gateway_id - ))), - "operator" => operator_allowlist - .clone() - .map(auth::k8s_sa::NamespaceValidator::Allowlist) - .ok_or_else(|| { - Error::config("Kubernetes operator namespace allowlist was not initialized") - }), - mode => Err(Error::config(format!( - "invalid Kubernetes workspace_mode '{mode}' for ServiceAccount bootstrap" - ))), - } -} - -fn validate_remote_compute_driver_config( - name: &str, - file: Option<&config_file::ConfigFile>, -) -> Result<()> { - if name != "kubernetes" - || !file.is_some_and(|file| file.openshell.drivers.contains_key("kubernetes")) - { - return Ok(()); - } - - let config = compute::driver_config::kubernetes_sa_bootstrap_config(file)?; - if config.workspace_mode == "operator" { - return Err(Error::config( - "Kubernetes workspace_mode 'operator' requires an in-process Kubernetes driver; \ - external Kubernetes compute drivers do not support operator mode", - )); - } - - Ok(()) -} - /// Opaque result returned by a compiled compute-driver factory. pub struct ComputeDriverBuildOutput { runtime: ComputeRuntime, @@ -1603,7 +1528,6 @@ async fn build_compute_runtime( (output.runtime, output.operator_allowlist) } ConfiguredComputeDriver::Remote { name } => { - validate_remote_compute_driver_config(&name, driver_startup.file)?; let remote_config = compute::driver_config::remote_driver_config_from_context(driver_startup, &name)?; info!( @@ -1768,7 +1692,6 @@ mod tests { allow_plaintext_service_http, bind_gateway_listeners, classify_initial_bytes, is_benign_tls_handshake_failure, kubernetes_sandbox_jwt_expiry_disabled, mint_gateway_extension_credential, serve_gateway_listener, - validate_remote_compute_driver_config, }; use openshell_core::{ ComputeDriverKind, Config, @@ -1805,28 +1728,6 @@ mod tests { ) } - #[test] - fn external_kubernetes_operator_workspace_mode_is_rejected() { - let file: crate::config_file::ConfigFile = toml::from_str( - r#" -[openshell.drivers.kubernetes] -socket_path = "/run/openshell/kubernetes.sock" -workspace_mode = "operator" -operator_namespace_label = "openshell.ai/workspace=true" -"#, - ) - .expect("valid config"); - - let error = validate_remote_compute_driver_config("kubernetes", Some(&file)) - .expect_err("external operator mode must fail closed"); - - assert!( - error - .to_string() - .contains("external Kubernetes compute drivers do not support operator mode") - ); - } - #[test] fn plaintext_extension_endpoint_is_rejected_unless_explicitly_opted_out() { let issuer = extension_test_issuer(); diff --git a/crates/openshell-server/src/multiplex.rs b/crates/openshell-server/src/multiplex.rs index 7a7125dcc3..d58a06c475 100644 --- a/crates/openshell-server/src/multiplex.rs +++ b/crates/openshell-server/src/multiplex.rs @@ -622,7 +622,7 @@ fn gateway_principal_fields(principal: &Principal) -> BTreeMap { match &sandbox.source { SandboxIdentitySource::BootstrapJwt { .. } => "bootstrap_jwt", SandboxIdentitySource::BootstrapCert { .. } => "bootstrap_cert", - SandboxIdentitySource::K8sServiceAccount { .. } => "k8s_service_account", + SandboxIdentitySource::ComputeDriver { .. } => "compute_driver", } .to_string(), ); @@ -853,10 +853,9 @@ where /// Assemble the authenticator chain for the gateway. /// /// Chain order (first-match-wins): -/// 1. `K8sServiceAccountAuthenticator` (path-scoped to `IssueSandboxToken`) -/// — exchanges a projected SA token for a `Principal::Sandbox` so the -/// `IssueSandboxToken` handler can mint a gateway JWT. No-op on every -/// other path; only present when the gateway runs in-cluster. +/// 1. `ComputeDriverAuthenticator` (path-scoped to `IssueSandboxToken`) +/// — delegates a driver-native credential and receives a sandbox identity +/// so the handler can mint a gateway JWT. No-op on every other path. /// 2. `SandboxJwtAuthenticator` — validates gateway-minted JWTs. Recognized /// via a distinctive `kid` so non-matching Bearer tokens fall through. /// 3. `OidcAuthenticator` — validates user Bearer tokens against the @@ -874,8 +873,8 @@ where /// to pass-through unless mTLS or local unauthenticated users are enabled. fn build_authenticator_chain(state: &ServerState) -> Option { let mut authenticators: Vec> = Vec::new(); - if let Some(k8s) = state.k8s_sa_authenticator.clone() { - authenticators.push(k8s); + if let Some(driver) = state.compute_driver_authenticator.clone() { + authenticators.push(driver); } if let Some(jwt) = state.sandbox_jwt_authenticator.clone() { authenticators.push(jwt); diff --git a/crates/openshell-server/src/test_support.rs b/crates/openshell-server/src/test_support.rs index 957db46fab..5f4063ae45 100644 --- a/crates/openshell-server/src/test_support.rs +++ b/crates/openshell-server/src/test_support.rs @@ -95,6 +95,7 @@ impl FakeComputeDriver { driver_version: "test".to_string(), default_image: "openshell/sandbox:test".to_string(), gateway_manages_lifecycle: false, + supports_sandbox_authentication: false, }, gateway_listener_requirements: Vec::new(), gateway_listener_requirements_supported: true, @@ -235,6 +236,16 @@ impl Stream for UnixIncoming { #[tonic::async_trait] impl ComputeDriver for FakeComputeDriver { + async fn authenticate_sandbox( + &self, + _request: Request, + ) -> Result, Status> + { + Err(Status::unimplemented( + "fake driver does not authenticate sandbox credentials", + )) + } + type WatchSandboxesStream = WatchStream; async fn get_capabilities( diff --git a/deploy/helm/openshell/ci/values-spire-stack.yaml b/deploy/helm/openshell/ci/values-spire-stack.yaml index b55f7cfc57..8a1e648829 100644 --- a/deploy/helm/openshell/ci/values-spire-stack.yaml +++ b/deploy/helm/openshell/ci/values-spire-stack.yaml @@ -15,7 +15,7 @@ spire-server: clusterSPIFFEIDs: openshell-sandboxes: enabled: true - spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/openshell/sandbox/{{ index .PodMeta.Annotations "openshell.io/sandbox-id" }}' + spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/openshell/sandbox/{{ index .PodMeta.Annotations "openshell.ai/sandbox-id" }}' namespaceSelector: matchLabels: kubernetes.io/metadata.name: openshell diff --git a/deploy/helm/openshell/templates/role.yaml b/deploy/helm/openshell/templates/role.yaml index af80989072..8def13f310 100644 --- a/deploy/helm/openshell/templates/role.yaml +++ b/deploy/helm/openshell/templates/role.yaml @@ -33,7 +33,7 @@ rules: - watch # Per-sandbox identity: TokenReview authenticates the projected token from # the configured sandbox service account, then the gateway resolves the - # returned pod name and UID to the pod's `openshell.io/sandbox-id` + # returned pod name and UID to the pod's `openshell.ai/sandbox-id` # annotation. patch is intentionally NOT granted — the annotation is set # once at pod create and must remain immutable for the lifetime of the # sandbox. diff --git a/docs/kubernetes/access-control.mdx b/docs/kubernetes/access-control.mdx index e3addad032..d6cb2140bd 100644 --- a/docs/kubernetes/access-control.mdx +++ b/docs/kubernetes/access-control.mdx @@ -21,11 +21,11 @@ For how the CLI resolves gateways and stores credentials, refer to [Gateway Auth ## Sandbox Supervisor Identity -Kubernetes sandbox supervisors authenticate back to the gateway as sandbox workloads. By default, the gateway mints its own sandbox JWTs and Kubernetes sandboxes bootstrap them with a projected ServiceAccount token. +Kubernetes sandbox supervisors authenticate back to the gateway as sandbox workloads. By default, the Kubernetes compute driver validates each projected ServiceAccount token and returns the authenticated sandbox ID to the gateway. The gateway verifies the sandbox still exists and mints its own sandbox JWT. Dynamic provider token grants can use SPIFFE without changing supervisor-to-gateway authentication. Set `server.providerTokenGrants.spiffe.enabled=true` to mount the SPIFFE CSI Workload API socket into gateway and sandbox pods while keeping the projected ServiceAccount token bootstrap and gateway-minted sandbox JWT path. -Provider token grants require a SPIFFE implementation such as SPIRE and identities for the gateway and sandbox pods. The repository's local SPIRE overlay assigns sandbox IDs from the pod's `openshell.io/sandbox-id` annotation, but the gateway validation path only requires the supervisor SVID to be valid and in the same SPIFFE trust domain as the gateway SVID. Provider profiles with `token_grant` metadata cause the sandbox supervisor to request JWT-SVIDs and exchange them for upstream OAuth2 access tokens. Token-exchange profiles also require a gateway SPIFFE identity because the gateway brokers the intermediate token exchange with its own JWT-SVID. +Provider token grants require a SPIFFE implementation such as SPIRE and identities for the gateway and sandbox pods. The repository's local SPIRE overlay assigns sandbox IDs from the pod's `openshell.ai/sandbox-id` annotation, but the gateway validation path only requires the supervisor SVID to be valid and in the same SPIFFE trust domain as the gateway SVID. Provider profiles with `token_grant` metadata cause the sandbox supervisor to request JWT-SVIDs and exchange them for upstream OAuth2 access tokens. Token-exchange profiles also require a gateway SPIFFE identity because the gateway brokers the intermediate token exchange with its own JWT-SVID. The gateway verifies supervisor JWT-SVIDs with JWT bundles fetched from the SPIFFE Workload API, so intermediate token exchange does not require gateway access to the SPIRE OIDC discovery endpoint or its TLS CA. diff --git a/docs/reference/gateway-auth.mdx b/docs/reference/gateway-auth.mdx index b5157b92a5..5969c43e64 100644 --- a/docs/reference/gateway-auth.mdx +++ b/docs/reference/gateway-auth.mdx @@ -191,7 +191,7 @@ to [Manage Workspaces and Access](/sandboxes/manage-workspaces). If `OPENSHELL_OIDC_SCOPES_CLAIM` is set, the gateway also enforces scopes. It accepts space-delimited scope strings such as `scope: "openid sandbox:read"` and JSON arrays such as `scp: ["sandbox:read"]`. Standard OIDC scopes such as `openid`, `profile`, `email`, and `offline_access` are ignored for authorization. `openshell:all` grants access to all scoped methods. -Supervisor-to-gateway RPCs do not use user OIDC tokens or mTLS user identity. Each sandbox supervisor presents a gateway-minted `Authorization: Bearer` token scoped to its sandbox ID. On Kubernetes, the gateway mints that token only after TokenReview validates the projected ServiceAccount token, the pod UID matches the live pod, and the pod's controlling `Sandbox` ownerReference matches the live Sandbox CR. Log upload, policy status, credential environment lookup, inference bundle lookup, and sandbox config sync run with sandbox-restricted scope, while CLI users authenticate with OIDC, edge auth, local mTLS user authentication, or an explicitly enabled unauthenticated local developer mode. `GetInferenceBundle` returns route material that includes provider credentials, so it requires a sandbox principal; user callers manage inference configuration through the user-facing inference APIs instead. +Supervisor-to-gateway RPCs do not use user OIDC tokens or mTLS user identity. Each sandbox supervisor presents a gateway-minted `Authorization: Bearer` token scoped to its sandbox ID. On Kubernetes, the Kubernetes compute driver validates the projected ServiceAccount token with TokenReview, verifies the live pod UID and controlling `Sandbox` ownerReference, and returns the authenticated sandbox ID to the gateway. The gateway verifies that sandbox still exists before minting its JWT. Log upload, policy status, credential environment lookup, inference bundle lookup, and sandbox config sync run with sandbox-restricted scope, while CLI users authenticate with OIDC, edge auth, local mTLS user authentication, or an explicitly enabled unauthenticated local developer mode. `GetInferenceBundle` returns route material that includes provider credentials, so it requires a sandbox principal; user callers manage inference configuration through the user-facing inference APIs instead. Re-authenticate an OIDC gateway with: diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index a066b14f01..97ce3f3324 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -250,6 +250,8 @@ gateway under the worktree-specific k3d cluster name; select it with `openshell gateway select `. The local Podman, Docker, and VM gateway tasks export to the forwarded receiver automatically. +In-process compute drivers read their backend-specific settings from `[openshell.drivers.]`. An external driver's gateway table supplies only its `socket_path`; configure the external driver process itself through that binary's flags or environment variables. A driver that advertises `supports_sandbox_authentication` may authenticate an opaque bootstrap credential through the compute-driver protocol. The gateway trusts the returned sandbox ID only for `IssueSandboxToken`, verifies that the sandbox still exists, and then mints its own JWT. The in-process Kubernetes driver reads `service_account_name`, `workspace_mode`, and namespace discovery from `[openshell.drivers.kubernetes]`; an external Kubernetes driver receives the equivalent values through its own CLI or environment contract. + ### Tuning This table decides whether and where to export. How the SDK exports is controlled by the standard OpenTelemetry environment variables, which the gateway reads through the SDK rather than mirroring as TOML keys: diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index 5773fb2bb1..47b761e5bd 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -373,7 +373,7 @@ For maintainer-level implementation details, refer to the [Kubernetes driver REA |---|---|---| | `compute_drivers = ["kubernetes"]` | Not applicable | Select the Kubernetes compute driver. | | `[openshell.drivers.kubernetes].namespace` | `server.sandboxNamespace` | Set the namespace for sandbox resources. The Helm chart defaults to the release namespace when left empty. | -| `service_account_name` | `sandboxServiceAccount.name` | Set the Kubernetes service account assigned to sandbox pods and accepted by the gateway TokenReview bootstrap path. The Helm chart creates a dedicated sandbox service account by default. | +| `service_account_name` | `sandboxServiceAccount.name` | Set the Kubernetes service account assigned to sandbox pods and accepted by the Kubernetes driver's TokenReview bootstrap path. The Helm chart creates a dedicated sandbox service account by default. | | `default_image` | `server.sandboxImage` | Set the default sandbox image. | | `image_pull_policy` | `server.sandboxImagePullPolicy` | Set the Kubernetes image pull policy for sandbox pods. | | `image_pull_secrets` | `server.sandboxImagePullSecrets` | Attach Kubernetes image-pull Secrets to sandbox pods. Managed mode copies these explicitly named Secrets from the configured source namespace into each workspace namespace. In shared and operator modes, the Secrets must already exist in the sandbox namespace. | diff --git a/proto/compute_driver.proto b/proto/compute_driver.proto index afa93f1b18..b7e972ddd4 100644 --- a/proto/compute_driver.proto +++ b/proto/compute_driver.proto @@ -23,6 +23,12 @@ service ComputeDriver { // Report driver capabilities and defaults. rpc GetCapabilities(GetCapabilitiesRequest) returns (GetCapabilitiesResponse); + // Authenticate a driver-native bootstrap credential and return the stable + // sandbox identity it represents. The gateway remains responsible for + // checking that the sandbox exists and minting gateway credentials. + rpc AuthenticateSandbox(AuthenticateSandboxRequest) + returns (AuthenticateSandboxResponse); + // Report additional gateway listeners required by this driver instance. // // A requirement is not authorization to expose the gateway. The gateway @@ -78,6 +84,18 @@ message GetCapabilitiesResponse { // Whether the gateway should stop running sandbox compute during graceful // shutdown and restart the retained running intent on startup. bool gateway_manages_lifecycle = 6; + // Whether AuthenticateSandbox is implemented by this driver. + bool supports_sandbox_authentication = 7; +} + +message AuthenticateSandboxRequest { + // Opaque credential whose format and verification are owned by the driver. + string credential = 1 [(openshell.options.v1.secret) = true]; +} + +message AuthenticateSandboxResponse { + // Stable gateway-assigned sandbox ID authenticated by the driver. + string sandbox_id = 1; } message GetGatewayListenerRequirementsRequest {}