From 89f06921f8c645dbfba3bbb65f6687026ee05e13 Mon Sep 17 00:00:00 2001 From: Alex Merced Date: Mon, 10 Aug 2026 09:45:41 -0400 Subject: [PATCH 01/23] fix(api): close the authorization bypasses and Iceberg conformance gaps Addresses the critical clusters of roadmap_aug10.md: the API-layer authorization bypasses (B0a-B0m), the Iceberg metadata conformance and commit-path defects (B11-B16o), and the middleware/reliability items. Authorization (pangolin_api) B0a POST /api/v1/tokens took no session at all: any authenticated principal could mint a Root JWT for any tenant. Now Root-only, or TenantAdmin within its own tenant and never above its rank. B0b Credential vending performed no authorization and hardcoded read+write for a table it never looked up. Now resolves the asset, requires Read, and vends write only when Write is held. B0c rename_table had no permission check. Now Write on the source and Create on the destination namespace, plus a 409 on collision. B0d update_namespace_properties discarded its session and never resolved the catalog. Now Write-scoped to the namespace. B0e create_view/get_view had no checks; a view's SQL is its whole definition. Now mirror create_table/load_table. B0f perform_maintenance ran destructive jobs against a hardcoded "default" catalog with no authz. Now uses the path catalog and requires Delete. B0g The Iceberg OAuth token endpoint checked `active` but not expiry, so an expired service user could renew indefinitely. B0h PANGOLIN_DEV_MODE waived the NO_AUTH public-bind guard - the two flags are routinely set together. The guard is unconditional. B0i PermissionScope::Tenant matched without comparing tenants, so a grant in tenant A satisfied resources in tenant B. B0j Logout revoked session.user_id, which no token carries as its jti, so tokens survived logout. UserSession now carries the jti; rotate_token revokes the rotated-out token too. B0k /api/v1/oauth/exchange was not public, making the OAuth login flow unreachable: the browser could never redeem its code. B0l OAuth linked accounts by unverified email. Identity is now (provider, subject); email linking needs a verified address and an operator domain allowlist. B0m expires_in_hours could panic token issuance before the checked arithmetic ran. Clamped, plus a CatchPanicLayer. B0o A malformed or absent jti skipped revocation entirely; the API-key branch also returned before the public-path check. Iceberg (pangolin_core, pangolin_api) B11-B14 default-spec-id (was current-partition-spec-id), last-partition-id, schema "type": "struct", and no more explicit nulls on optional fields. B13 metadata-log is appended on every commit and truncated to write.metadata.previous-versions-max. B15 A client sequence number can no longer jump the counter to i64::MAX and overflow the next commit. B16 A feature-branch commit no longer moves main. B16a One shared parse_namespace across all handlers: a nested namespace registered on create is now found on commit. B16b last-updated-ms advances on every commit, not just snapshots. B16c -1 resolves against what this commit added, not vec.last(); duplicate Add* ids are rejected. B16d/g Metadata files are written before registration and reclaimed on a lost CAS instead of orphaned. B16e create_table returned the table directory as metadata-location. B16f The hand-rolled schema parser dropped complex columns, forced every field optional and widened int to long. Deserialized now. B16h Namespace property removals were silently ignored. B16i pageToken/pageSize and next-page-token on list responses. B16j Iceberg handlers return the spec error envelope. B16k Federated forwarding on create/delete namespace and the tree. B16l The timeout layer sits outside the concurrency limiter, so queued requests have a deadline. B16m delete_warehouse deletes before invalidating, closing the window where a racing read re-cached deleted credentials. B16n PANGOLIN_SHUTDOWN_GRACE_SECS actually bounds the drain. Storage B17 One ns_key helper on the memory backend; multi-level namespaces were undeletable there. New CatalogStore::delete_file and replace_namespace_properties, implemented across all four backends. Also completes loadNamespaceMetadata and namespaceExists. cargo test --workspace: 57 targets green. Co-Authored-By: Claude Opus 5 --- pangolin/Cargo.lock | 1 + pangolin/Cargo.toml | 2 +- pangolin/pangolin_api/src/asset_handlers.rs | 97 ++- pangolin/pangolin_api/src/auth.rs | 8 + pangolin/pangolin_api/src/auth_middleware.rs | 47 +- pangolin/pangolin_api/src/authz.rs | 2 + pangolin/pangolin_api/src/authz_utils.rs | 168 +++-- .../src/business_metadata_handlers.rs | 2 + pangolin/pangolin_api/src/cached_store.rs | 34 +- pangolin/pangolin_api/src/config.rs | 25 +- .../pangolin_api/src/dashboard_handlers.rs | 3 + pangolin/pangolin_api/src/iceberg/commit.rs | 264 +++++-- pangolin/pangolin_api/src/iceberg/error.rs | 41 ++ pangolin/pangolin_api/src/iceberg/mod.rs | 5 +- .../pangolin_api/src/iceberg/namespaces.rs | 452 ++++++++++-- pangolin/pangolin_api/src/iceberg/oauth.rs | 12 +- pangolin/pangolin_api/src/iceberg/tables.rs | 657 ++++++++++++------ pangolin/pangolin_api/src/iceberg/types.rs | 112 +++ pangolin/pangolin_api/src/lib.rs | 36 +- pangolin/pangolin_api/src/main.rs | 30 +- pangolin/pangolin_api/src/oauth_handlers.rs | 79 ++- .../pangolin_api/src/optimization_handlers.rs | 21 +- .../pangolin_api/src/pangolin_handlers.rs | 8 +- pangolin/pangolin_api/src/public_paths.rs | 27 + pangolin/pangolin_api/src/signing_handlers.rs | 84 ++- pangolin/pangolin_api/src/token_handlers.rs | 233 +++++-- .../tests/business_metadata_test.rs | 1 + .../tests/credential_vending_tests.rs | 2 + .../tests/iceberg_handlers_test.rs | 3 + .../tests/signing_handlers_test.rs | 2 + .../pangolin_core/src/iceberg_metadata.rs | 113 +++ pangolin/pangolin_core/src/user.rs | 8 + pangolin/pangolin_store/src/file_delete.rs | 64 ++ pangolin/pangolin_store/src/lib.rs | 22 + pangolin/pangolin_store/src/memory/mod.rs | 20 + .../pangolin_store/src/memory/namespaces.rs | 37 +- pangolin/pangolin_store/src/mongo/mod.rs | 19 + .../pangolin_store/src/mongo/namespaces.rs | 28 + pangolin/pangolin_store/src/postgres/main.rs | 20 + .../pangolin_store/src/postgres/namespaces.rs | 22 + pangolin/pangolin_store/src/sqlite/main.rs | 19 + .../pangolin_store/src/sqlite/namespaces.rs | 28 + roadmap_aug10.md | 423 +++++++++++ 43 files changed, 2761 insertions(+), 520 deletions(-) create mode 100644 pangolin/pangolin_store/src/file_delete.rs create mode 100644 roadmap_aug10.md diff --git a/pangolin/Cargo.lock b/pangolin/Cargo.lock index 57cd327..84f1ea0 100644 --- a/pangolin/Cargo.lock +++ b/pangolin/Cargo.lock @@ -5522,6 +5522,7 @@ checksum = "1e9cd434a998747dd2c4276bc96ee2e0c7a2eadf3cae88e52be55a05fa9053f5" dependencies = [ "bitflags 2.10.0", "bytes", + "futures-util", "http 1.4.0", "http-body 1.0.1", "http-body-util", diff --git a/pangolin/Cargo.toml b/pangolin/Cargo.toml index 1ca930d..08b9c0d 100644 --- a/pangolin/Cargo.toml +++ b/pangolin/Cargo.toml @@ -63,7 +63,7 @@ async-trait = "0.1" dashmap = "6.0" futures = "0.3" tower = { version = "0.4", features = ["limit", "util"] } -tower-http = { version = "0.5", features = ["cors", "trace", "timeout", "limit"] } +tower-http = { version = "0.5", features = ["cors", "trace", "timeout", "limit", "catch-panic"] } reqwest = { version = "0.11", features = ["json"] } url = "2.5" bytes = "1.5" diff --git a/pangolin/pangolin_api/src/asset_handlers.rs b/pangolin/pangolin_api/src/asset_handlers.rs index 9e378f5..3128f67 100644 --- a/pangolin/pangolin_api/src/asset_handlers.rs +++ b/pangolin/pangolin_api/src/asset_handlers.rs @@ -1,7 +1,7 @@ use crate::auth::TenantId; use crate::authz::check_permission; -use crate::iceberg::parse_table_identifier; use crate::iceberg::AppState; +use crate::iceberg::{parse_namespace, parse_table_identifier}; use axum::{ extract::{Extension, Path, Query, State}, http::StatusCode, @@ -66,6 +66,7 @@ impl From for ViewResponse { pub async fn create_view( State(store): State, Extension(tenant): Extension, + Extension(session): Extension, Path((prefix, namespace)): Path<(String, String)>, Json(payload): Json, ) -> impl IntoResponse { @@ -73,10 +74,35 @@ pub async fn create_view( let catalog_name = prefix; let (view_name, branch_from_name) = parse_table_identifier(&payload.name); - let branch = branch_from_name.unwrap_or("main".to_string()); + let (namespace_parts, branch_from_ns) = parse_namespace(&namespace); + let branch = branch_from_name + .or(branch_from_ns) + .unwrap_or_else(|| "main".to_string()); - // Parse namespace - let namespace_parts: Vec = namespace.split('\x1F').map(|s| s.to_string()).collect(); + let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { + Ok(Some(c)) => c, + Ok(None) => return (StatusCode::NOT_FOUND, "Catalog not found").into_response(), + Err(e) => { + tracing::error!(error = %e, "create_view: failed to load catalog"); + return (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response(); + } + }; + + // B0e: this handler took no session and performed no authorization at all, + // so any tenant member could create a view in any namespace. It now mirrors + // `create_table`'s namespace-scoped Create check. + let scope = PermissionScope::Namespace { + catalog_id: catalog.id, + namespace: namespace_parts.join("."), + }; + match check_permission(&store, &session, &Action::Create, &scope).await { + Ok(true) => (), + Ok(false) => return (StatusCode::FORBIDDEN, "Forbidden").into_response(), + Err(e) => { + tracing::error!(error = %e, "create_view: permission check failed"); + return (StatusCode::INTERNAL_SERVER_ERROR, "Permission check failed").into_response(); + } + } let mut properties = payload.properties.unwrap_or_default(); properties.insert("sql".to_string(), payload.sql); @@ -104,7 +130,10 @@ pub async fn create_view( .await { Ok(_) => (StatusCode::CREATED, Json(ViewResponse::from(asset))).into_response(), - Err(_) => (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response(), + Err(e) => { + tracing::error!(error = %e, "create_view: failed to create asset"); + (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response() + } } } @@ -128,35 +157,64 @@ pub async fn create_view( pub async fn get_view( State(store): State, Extension(tenant): Extension, + Extension(session): Extension, Path((prefix, namespace, view)): Path<(String, String, String)>, ) -> impl IntoResponse { let tenant_id = tenant.0; let catalog_name = prefix; let (view_name, branch_from_name) = parse_table_identifier(&view); - let branch = branch_from_name.unwrap_or("main".to_string()); + let (namespace_parts, branch_from_ns) = parse_namespace(&namespace); + let branch = branch_from_name + .or(branch_from_ns) + .unwrap_or_else(|| "main".to_string()); - let namespace_parts: Vec = namespace.split('\x1F').map(|s| s.to_string()).collect(); + let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { + Ok(Some(c)) => c, + Ok(None) => return (StatusCode::NOT_FOUND, "Catalog not found").into_response(), + Err(e) => { + tracing::error!(error = %e, "get_view: failed to load catalog"); + return (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response(); + } + }; - match store + let asset = match store .get_asset( tenant_id, &catalog_name, Some(branch), - namespace_parts, + namespace_parts.clone(), view_name, ) .await { - Ok(Some(asset)) => { - if asset.kind == AssetType::View { - (StatusCode::OK, Json(ViewResponse::from(asset))).into_response() - } else { - (StatusCode::NOT_FOUND, "Asset is not a view").into_response() - } + Ok(Some(a)) => a, + Ok(None) => return (StatusCode::NOT_FOUND, "View not found").into_response(), + Err(e) => { + tracing::error!(error = %e, "get_view: failed to load asset"); + return (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response(); + } + }; + + if asset.kind != AssetType::View { + return (StatusCode::NOT_FOUND, "Asset is not a view").into_response(); + } + + // B0e: a view's `properties["sql"]` is its whole definition - business + // logic, column names, sometimes filter predicates that reveal the data. + // Reading it used to require nothing beyond being authenticated. + let scope = PermissionScope::Asset { + catalog_id: catalog.id, + namespace: namespace_parts.join("."), + asset_id: asset.id, + }; + match check_permission(&store, &session, &Action::Read, &scope).await { + Ok(true) => (StatusCode::OK, Json(ViewResponse::from(asset))).into_response(), + Ok(false) => (StatusCode::FORBIDDEN, "Forbidden").into_response(), + Err(e) => { + tracing::error!(error = %e, "get_view: permission check failed"); + (StatusCode::INTERNAL_SERVER_ERROR, "Permission check failed").into_response() } - Ok(None) => (StatusCode::NOT_FOUND, "View not found").into_response(), - Err(_) => (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response(), } } @@ -487,6 +545,7 @@ pub async fn list_assets( // Filter assets based on permissions let filtered = crate::authz_utils::filter_assets( + tenant_id, assets_with_metadata, &permissions, session.role, @@ -613,6 +672,7 @@ mod tests { tenant_id: Some(tenant_id), expires_at: chrono::Utc::now() + chrono::Duration::hours(1), issued_at: chrono::Utc::now(), + token_id: None, }; let payload = RegisterAssetRequest { @@ -646,6 +706,7 @@ mod tests { tenant_id: Some(tenant_id), expires_at: chrono::Utc::now() + chrono::Duration::hours(1), issued_at: chrono::Utc::now(), + token_id: None, }; let payload = RegisterAssetRequest { @@ -680,6 +741,7 @@ mod tests { tenant_id: Some(tenant_id), expires_at: chrono::Utc::now() + chrono::Duration::hours(1), issued_at: chrono::Utc::now(), + token_id: None, }; let payload = RegisterAssetRequest { @@ -713,6 +775,7 @@ mod tests { tenant_id: Some(tenant_id), expires_at: chrono::Utc::now() + chrono::Duration::hours(1), issued_at: chrono::Utc::now(), + token_id: None, }; let asset_types = vec![ diff --git a/pangolin/pangolin_api/src/auth.rs b/pangolin/pangolin_api/src/auth.rs index 50592a3..b13700f 100644 --- a/pangolin/pangolin_api/src/auth.rs +++ b/pangolin/pangolin_api/src/auth.rs @@ -47,6 +47,14 @@ impl Claims { .ok_or("Invalid issued_at timestamp")?, expires_at: DateTime::from_timestamp(self.exp, 0) .ok_or("Invalid expires_at timestamp")?, + // Carried through so logout can revoke the token that is actually + // presented rather than the user id (B0j). + token_id: self + .jti + .as_ref() + .map(|id| Uuid::parse_str(id)) + .transpose() + .map_err(|e| e.to_string())?, }) } } diff --git a/pangolin/pangolin_api/src/auth_middleware.rs b/pangolin/pangolin_api/src/auth_middleware.rs index b8f3a65..4ca76eb 100644 --- a/pangolin/pangolin_api/src/auth_middleware.rs +++ b/pangolin/pangolin_api/src/auth_middleware.rs @@ -57,6 +57,12 @@ pub fn create_session( role, issued_at: now, expires_at, + // Sessions built here are not derived from a presented JWT: they are + // either about to have a token minted *from* them (login), or backed by + // an API key / root basic auth, which have nothing to revoke. Sessions + // that do come from a bearer token get their `jti` in + // `Claims::to_session`. + token_id: None, } } @@ -223,6 +229,18 @@ pub async fn auth_middleware( // tests and tools can exercise a specific identity even in NO_AUTH mode. } + // Public endpoints, matched structurally rather than by suffix (A-11). + // + // B0o: this check used to sit *below* the API-key branch, which returned + // unconditionally. A client that sets `X-API-Key` globally - the normal way + // to configure an HTTP client - therefore could not reach `/v1/config`, + // `/health` or the OAuth token endpoint at all, the opposite of the + // documented ordering. Resolving public paths first also keeps an + // unauthenticated `/health` probe away from bcrypt. + if crate::public_paths::is_public_path(&path) { + return next.run(req).await; + } + // Service-user API key. if let Some(api_key_header) = req.headers().get("X-API-Key") { let Ok(api_key) = api_key_header.to_str() else { @@ -271,11 +289,6 @@ pub async fn auth_middleware( } } - // Public endpoints, matched structurally rather than by suffix (A-11). - if crate::public_paths::is_public_path(&path) { - return next.run(req).await; - } - let auth_header = req .headers() .get(header::AUTHORIZATION) @@ -369,8 +382,19 @@ pub async fn auth_middleware( // Revocation must fail *closed*. Previously an error from the store was // logged and ignored, so during a database blip every revoked token was // accepted again (A-13). - if let Some(ref jti_str) = claims.jti { - if let Ok(token_id) = uuid::Uuid::parse_str(jti_str) { + // + // B0o: the two nested `if let`s also failed open on the *shape* of the + // claim. A token whose `jti` was present but not a UUID skipped the + // revocation check entirely and was unrevocable for its whole lifetime, and + // a token minted with no `jti` at all was likewise exempt. A malformed `jti` + // is now a hard rejection, and the no-`jti` case is accepted only when the + // operator has explicitly opted into legacy tokens. + match claims.jti.as_deref() { + Some(jti_str) => { + let Ok(token_id) = uuid::Uuid::parse_str(jti_str) else { + tracing::warn!(jti = %jti_str, "rejected a token with a malformed jti"); + return (StatusCode::UNAUTHORIZED, "Invalid token").into_response(); + }; match store.is_token_revoked(token_id).await { Ok(true) => { tracing::warn!(jti = %jti_str, "revoked token presented"); @@ -390,6 +414,15 @@ pub async fn auth_middleware( } } } + None => { + if !crate::config::allow_tokens_without_jti() { + tracing::warn!( + "rejected a token with no jti; it could never be revoked. Re-issue the \ + token, or set PANGOLIN_ALLOW_TOKENS_WITHOUT_JTI=true during migration" + ); + return (StatusCode::UNAUTHORIZED, "Invalid token").into_response(); + } + } } let session = match claims.to_session() { diff --git a/pangolin/pangolin_api/src/authz.rs b/pangolin/pangolin_api/src/authz.rs index adcc747..a3397f0 100644 --- a/pangolin/pangolin_api/src/authz.rs +++ b/pangolin/pangolin_api/src/authz.rs @@ -183,6 +183,7 @@ mod tests { username: "test_user".to_string(), issued_at: chrono::Utc::now(), expires_at: chrono::Utc::now() + chrono::Duration::hours(1), + token_id: None, }; // Target Action/Scope @@ -267,6 +268,7 @@ mod tests { username: "test_user".to_string(), issued_at: chrono::Utc::now(), expires_at: chrono::Utc::now() + chrono::Duration::hours(1), + token_id: None, }; // --- Branch Permission Test --- diff --git a/pangolin/pangolin_api/src/authz_utils.rs b/pangolin/pangolin_api/src/authz_utils.rs index 999f5fd..c2abfec 100644 --- a/pangolin/pangolin_api/src/authz_utils.rs +++ b/pangolin/pangolin_api/src/authz_utils.rs @@ -3,12 +3,26 @@ use pangolin_core::permission::{Action, Permission, PermissionScope}; use pangolin_core::user::UserRole; use uuid::Uuid; +/// Does a `Tenant`-scoped grant apply to a resource in `resource_tenant_id`? +/// +/// B0i: all three access checks below used a bare `PermissionScope::Tenant => +/// true`, never comparing the grant's own `tenant_id` against the resource's. +/// A tenant-wide grant issued in tenant A therefore satisfied access checks for +/// resources in tenant B. Nothing exploited it today only because callers +/// pre-scope their store queries to one tenant - but every path that can +/// surface cross-tenant rows (root impersonation, search, dashboards) leaked +/// through it, and the invariant was one refactor away from mattering. +fn tenant_grant_applies(perm: &Permission, resource_tenant_id: Uuid) -> bool { + perm.tenant_id == resource_tenant_id +} + /// Check if a user has access to a catalog based on their permissions /// /// Checks for Read or Discoverable actions on: /// - Exact catalog scope -/// - Tenant-wide scope +/// - Tenant-wide scope, when the grant belongs to the resource's tenant pub fn has_catalog_access( + resource_tenant_id: Uuid, catalog_id: Uuid, permissions: &[Permission], required_actions: &[Action], @@ -18,7 +32,8 @@ pub fn has_catalog_access( let scope_matches = matches!( &perm.scope, PermissionScope::Catalog { catalog_id: cid } if *cid == catalog_id - ) || matches!(&perm.scope, PermissionScope::Tenant); + ) || (matches!(&perm.scope, PermissionScope::Tenant) + && tenant_grant_applies(perm, resource_tenant_id)); // Check if permission has any of the required actions let has_action = required_actions @@ -34,8 +49,9 @@ pub fn has_catalog_access( /// Checks for Read or Discoverable actions on: /// - Exact namespace scope /// - Parent catalog scope -/// - Tenant-wide scope +/// - Tenant-wide scope, when the grant belongs to the resource's tenant pub fn has_namespace_access( + resource_tenant_id: Uuid, catalog_id: Uuid, namespace: &str, permissions: &[Permission], @@ -49,7 +65,7 @@ pub fn has_namespace_access( namespace: ns, } => *cid == catalog_id && ns == namespace, PermissionScope::Catalog { catalog_id: cid } => *cid == catalog_id, - PermissionScope::Tenant => true, + PermissionScope::Tenant => tenant_grant_applies(perm, resource_tenant_id), _ => false, }; @@ -68,8 +84,9 @@ pub fn has_namespace_access( /// - Exact asset scope /// - Parent namespace scope /// - Parent catalog scope -/// - Tenant-wide scope +/// - Tenant-wide scope, when the grant belongs to the resource's tenant pub fn has_asset_access( + resource_tenant_id: Uuid, catalog_id: Uuid, namespace: &str, asset_id: Uuid, @@ -89,7 +106,7 @@ pub fn has_asset_access( namespace: ns, } => *cid == catalog_id && ns == namespace, PermissionScope::Catalog { catalog_id: cid } => *cid == catalog_id, - PermissionScope::Tenant => true, + PermissionScope::Tenant => tenant_grant_applies(perm, resource_tenant_id), _ => false, }; @@ -107,6 +124,7 @@ pub fn has_asset_access( /// Returns only catalogs the user has Read or Discoverable access to. /// Root and TenantAdmin users bypass filtering. pub fn filter_catalogs( + resource_tenant_id: Uuid, catalogs: Vec, permissions: &[Permission], user_role: UserRole, @@ -120,7 +138,14 @@ pub fn filter_catalogs( catalogs .into_iter() - .filter(|catalog| has_catalog_access(catalog.id, permissions, &required_actions)) + .filter(|catalog| { + has_catalog_access( + resource_tenant_id, + catalog.id, + permissions, + &required_actions, + ) + }) .collect() } @@ -129,6 +154,7 @@ pub fn filter_catalogs( /// Returns only namespaces the user has Read or Discoverable access to. /// Root and TenantAdmin users bypass filtering. pub fn filter_namespaces( + resource_tenant_id: Uuid, namespaces: Vec<(Namespace, String)>, permissions: &[Permission], user_role: UserRole, @@ -147,7 +173,13 @@ pub fn filter_namespaces( // Get catalog ID from the map if let Some(&catalog_id) = catalog_id_map.get(catalog_name) { let namespace_str = namespace.name.join("."); - has_namespace_access(catalog_id, &namespace_str, permissions, &required_actions) + has_namespace_access( + resource_tenant_id, + catalog_id, + &namespace_str, + permissions, + &required_actions, + ) } else { false } @@ -160,6 +192,7 @@ pub fn filter_namespaces( /// Returns only assets the user has Read or Discoverable access to. /// Root and TenantAdmin users bypass filtering. pub fn filter_assets( + resource_tenant_id: Uuid, assets: Vec<( Asset, Option, @@ -196,6 +229,7 @@ pub fn filter_assets( if let Some(&catalog_id) = catalog_id_map.get(catalog_name) { let namespace_str = namespace.join("."); has_asset_access( + resource_tenant_id, catalog_id, &namespace_str, asset.id, @@ -215,23 +249,29 @@ mod tests { use chrono::Utc; use std::collections::HashSet; - #[test] - fn test_has_catalog_access_with_catalog_permission() { - let catalog_id = Uuid::new_v4(); + /// Build a permission with a single `Read` action. + fn read_permission(tenant_id: Uuid, scope: PermissionScope) -> Permission { let mut actions = HashSet::new(); actions.insert(Action::Read); - - let permission = Permission { + Permission { id: Uuid::new_v4(), user_id: Uuid::new_v4(), - tenant_id: Uuid::new_v4(), - scope: PermissionScope::Catalog { catalog_id }, + tenant_id, + scope, actions, granted_by: Uuid::new_v4(), granted_at: Utc::now(), - }; + } + } + + #[test] + fn test_has_catalog_access_with_catalog_permission() { + let tenant_id = Uuid::new_v4(); + let catalog_id = Uuid::new_v4(); + let permission = read_permission(tenant_id, PermissionScope::Catalog { catalog_id }); assert!(has_catalog_access( + tenant_id, catalog_id, &[permission], &[Action::Read] @@ -240,47 +280,70 @@ mod tests { #[test] fn test_has_catalog_access_with_tenant_permission() { + let tenant_id = Uuid::new_v4(); let catalog_id = Uuid::new_v4(); - let mut actions = HashSet::new(); - actions.insert(Action::Read); - - let permission = Permission { - id: Uuid::new_v4(), - user_id: Uuid::new_v4(), - tenant_id: Uuid::new_v4(), - scope: PermissionScope::Tenant, - actions, - granted_by: Uuid::new_v4(), - granted_at: Utc::now(), - }; + let permission = read_permission(tenant_id, PermissionScope::Tenant); assert!(has_catalog_access( + tenant_id, catalog_id, &[permission], &[Action::Read] )); } + /// Regression test for B0i: a `Tenant`-scoped grant issued in tenant A must + /// not satisfy access for a resource in tenant B. + #[test] + fn tenant_scoped_grant_does_not_cross_tenants() { + let tenant_a = Uuid::new_v4(); + let tenant_b = Uuid::new_v4(); + let catalog_id = Uuid::new_v4(); + let asset_id = Uuid::new_v4(); + let permission = read_permission(tenant_a, PermissionScope::Tenant); + let grants = [permission]; + + assert!(has_catalog_access( + tenant_a, + catalog_id, + &grants, + &[Action::Read] + )); + assert!( + !has_catalog_access(tenant_b, catalog_id, &grants, &[Action::Read]), + "a tenant-A grant must not authorize a tenant-B catalog" + ); + assert!( + !has_namespace_access(tenant_b, catalog_id, "sales", &grants, &[Action::Read]), + "a tenant-A grant must not authorize a tenant-B namespace" + ); + assert!( + !has_asset_access( + tenant_b, + catalog_id, + "sales", + asset_id, + &grants, + &[Action::Read] + ), + "a tenant-A grant must not authorize a tenant-B asset" + ); + } + #[test] fn test_has_catalog_access_without_permission() { + let tenant_id = Uuid::new_v4(); let catalog_id = Uuid::new_v4(); let other_catalog_id = Uuid::new_v4(); - let mut actions = HashSet::new(); - actions.insert(Action::Read); - - let permission = Permission { - id: Uuid::new_v4(), - user_id: Uuid::new_v4(), - tenant_id: Uuid::new_v4(), - scope: PermissionScope::Catalog { + let permission = read_permission( + tenant_id, + PermissionScope::Catalog { catalog_id: other_catalog_id, }, - actions, - granted_by: Uuid::new_v4(), - granted_at: Utc::now(), - }; + ); assert!(!has_catalog_access( + tenant_id, catalog_id, &[permission], &[Action::Read] @@ -299,7 +362,7 @@ mod tests { properties: std::collections::HashMap::new(), }]; - let filtered = filter_catalogs(catalogs.clone(), &[], UserRole::Root); + let filtered = filter_catalogs(Uuid::new_v4(), catalogs.clone(), &[], UserRole::Root); assert_eq!(filtered.len(), catalogs.len()); } @@ -316,20 +379,15 @@ mod tests { properties: std::collections::HashMap::new(), }]; - let mut actions = HashSet::new(); - actions.insert(Action::Read); - - let permission = Permission { - id: Uuid::new_v4(), - user_id: Uuid::new_v4(), - tenant_id: Uuid::new_v4(), - scope: PermissionScope::Catalog { catalog_id }, - actions, - granted_by: Uuid::new_v4(), - granted_at: Utc::now(), - }; + let tenant_id = Uuid::new_v4(); + let permission = read_permission(tenant_id, PermissionScope::Catalog { catalog_id }); - let filtered = filter_catalogs(catalogs.clone(), &[permission], UserRole::TenantUser); + let filtered = filter_catalogs( + tenant_id, + catalogs.clone(), + &[permission], + UserRole::TenantUser, + ); assert_eq!(filtered.len(), 1); } @@ -346,7 +404,7 @@ mod tests { properties: std::collections::HashMap::new(), }]; - let filtered = filter_catalogs(catalogs, &[], UserRole::TenantUser); + let filtered = filter_catalogs(Uuid::new_v4(), catalogs, &[], UserRole::TenantUser); assert_eq!(filtered.len(), 0); } } diff --git a/pangolin/pangolin_api/src/business_metadata_handlers.rs b/pangolin/pangolin_api/src/business_metadata_handlers.rs index 1b41c7e..fe96955 100644 --- a/pangolin/pangolin_api/src/business_metadata_handlers.rs +++ b/pangolin/pangolin_api/src/business_metadata_handlers.rs @@ -243,6 +243,7 @@ pub async fn search_assets( // Apply permission-based filtering let filtered_results = crate::authz_utils::filter_assets( + tenant_id, results, &permissions, session.role.clone(), @@ -262,6 +263,7 @@ pub async fn search_assets( let ns_str = namespace.join("."); let required_actions = vec![pangolin_core::permission::Action::Read]; crate::authz_utils::has_asset_access( + tenant_id, catalog_id, &ns_str, asset.id, diff --git a/pangolin/pangolin_api/src/cached_store.rs b/pangolin/pangolin_api/src/cached_store.rs index ff99fed..52a36e0 100644 --- a/pangolin/pangolin_api/src/cached_store.rs +++ b/pangolin/pangolin_api/src/cached_store.rs @@ -101,16 +101,25 @@ impl CatalogStore for CachedCatalogStore { } async fn delete_warehouse(&self, tenant_id: Uuid, name: String) -> Result<()> { - // Invalidate cache on delete + // B16m: delete first, *then* invalidate. Invalidating first opened a + // window in which a concurrent `get_warehouse` missed the cache, read the + // still-present row, and re-inserted it with a full TTL - so the deleted + // warehouse's cloud credentials kept being vended for up to the TTL after + // delete returned success. Deleting first means any racing read either + // sees the row (and the invalidate below clears it) or does not find it. + let key = (tenant_id, name.clone()); + let result = self.inner.delete_warehouse(tenant_id, name.clone()).await; + tracing::info!( "Cache INVALIDATE (Delete) for warehouse: {}/{}", tenant_id, name ); - self.warehouse_cache - .invalidate(&(tenant_id, name.clone())) - .await; - self.inner.delete_warehouse(tenant_id, name).await + // Invalidate on the error path too: the delete may have partially + // applied, and a stale credential entry is the worse failure. + self.warehouse_cache.invalidate(&key).await; + + result } // --- Passthrough Operations (Uncached) --- @@ -224,6 +233,18 @@ impl CatalogStore for CachedCatalogStore { .await } + async fn replace_namespace_properties( + &self, + tenant_id: Uuid, + catalog_name: &str, + namespace: Vec, + properties: std::collections::HashMap, + ) -> Result<()> { + self.inner + .replace_namespace_properties(tenant_id, catalog_name, namespace, properties) + .await + } + // Asset async fn create_asset( &self, @@ -476,6 +497,9 @@ impl CatalogStore for CachedCatalogStore { async fn write_file(&self, location: &str, content: Vec) -> Result<()> { self.inner.write_file(location, content).await } + async fn delete_file(&self, location: &str) -> Result<()> { + self.inner.delete_file(location).await + } // Maintenance Operations async fn expire_snapshots( diff --git a/pangolin/pangolin_api/src/config.rs b/pangolin/pangolin_api/src/config.rs index 919bcbb..b08649d 100644 --- a/pangolin/pangolin_api/src/config.rs +++ b/pangolin/pangolin_api/src/config.rs @@ -102,6 +102,13 @@ pub struct AppConfig { pub oauth_redirect_allowlist: Vec, /// Whether API keys minted before the key-id format are still accepted. pub allow_legacy_api_keys: bool, + /// Whether JWTs carrying no `jti` are still accepted (off by default). + /// + /// `Claims.jti` is `Option` "for compatibility", and the middleware + /// used to skip the revocation check for such tokens - making them + /// unrevocable for their full lifetime (B0o). They are now rejected unless + /// an operator opts in for a migration window. + pub allow_tokens_without_jti: bool, /// Bind address and port. pub bind_address: String, pub port: u16, @@ -213,7 +220,14 @@ impl AppConfig { let bind_address = env_opt("PANGOLIN_BIND_ADDRESS").unwrap_or_else(|| "0.0.0.0".to_string()); - if no_auth && !dev_mode && !is_loopback(&bind_address) { + // B0h: the guard used to read `no_auth && !dev_mode && !is_loopback(..)`. + // That `!dev_mode` term meant `PANGOLIN_NO_AUTH=true PANGOLIN_DEV_MODE=true` + // started happily on the default `0.0.0.0` bind and treated every + // anonymous request as `TenantAdmin` - and those two flags are routinely + // set together in compose and dev setups, so the escape hatch was the + // common case. Dev mode relaxes secret strength, never network exposure: + // if auth is off, the listener must be loopback, unconditionally. + if no_auth && !is_loopback(&bind_address) { return Err(ConfigError::NoAuthOnPublicBind(bind_address)); } @@ -250,6 +264,7 @@ impl AppConfig { frontend_url, oauth_redirect_allowlist, allow_legacy_api_keys: env_bool("PANGOLIN_ALLOW_LEGACY_API_KEYS"), + allow_tokens_without_jti: env_bool("PANGOLIN_ALLOW_TOKENS_WITHOUT_JTI"), bind_address, port: env_parsed("PORT", 8080u16)?, log_format, @@ -342,6 +357,14 @@ pub fn allow_legacy_api_keys() -> bool { env_bool("PANGOLIN_ALLOW_LEGACY_API_KEYS") } +/// Whether JWTs with no `jti` are still honoured (off by default, see B0o). +pub fn allow_tokens_without_jti() -> bool { + if let Some(cfg) = CONFIG.get() { + return cfg.allow_tokens_without_jti; + } + env_bool("PANGOLIN_ALLOW_TOKENS_WITHOUT_JTI") +} + /// Constant-time string comparison, for credential checks. /// /// `==` on `&str` short-circuits on the first differing byte and therefore diff --git a/pangolin/pangolin_api/src/dashboard_handlers.rs b/pangolin/pangolin_api/src/dashboard_handlers.rs index 02f81b6..f43b4e1 100644 --- a/pangolin/pangolin_api/src/dashboard_handlers.rs +++ b/pangolin/pangolin_api/src/dashboard_handlers.rs @@ -132,6 +132,7 @@ pub async fn get_dashboard_stats( .await .map_err(ApiError::from)?; let accessible_catalogs = crate::authz_utils::filter_catalogs( + tenant_id, all_catalogs, &permissions, session.role.clone(), @@ -172,6 +173,7 @@ pub async fn get_dashboard_stats( .collect(); let filtered = crate::authz_utils::filter_namespaces( + tenant_id, namespace_tuples, &permissions, session.role.clone(), @@ -186,6 +188,7 @@ pub async fn get_dashboard_stats( let mut accessible_tables_count = 0; if let Ok(all_assets) = store.search_assets(tenant_id, "", None).await { let filtered_assets = crate::authz_utils::filter_assets( + tenant_id, all_assets, &permissions, session.role.clone(), diff --git a/pangolin/pangolin_api/src/iceberg/commit.rs b/pangolin/pangolin_api/src/iceberg/commit.rs index 1ff21b3..9357ac2 100644 --- a/pangolin/pangolin_api/src/iceberg/commit.rs +++ b/pangolin/pangolin_api/src/iceberg/commit.rs @@ -207,14 +207,22 @@ pub fn check_requirements( } } - // Pangolin does not track partition-field id assignment separately - // from the specs themselves, so this cannot be verified. Refusing - // is the honest answer: the alternative is pretending a - // precondition held when it was never examined. - CommitRequirement::AssertLastAssignedPartitionId { .. } => { - return Err(CommitError::Unsupported { - operation: "assert-last-assigned-partition-id".into(), - }) + // Now checkable: `last-partition-id` is a real field on + // `TableMetadata` (B12). Before it existed this requirement had to + // be refused, because the alternative was pretending a precondition + // held when it had never been examined. + CommitRequirement::AssertLastAssignedPartitionId { + last_assigned_partition_id, + } => { + if metadata.last_partition_id != *last_assigned_partition_id { + return Err(CommitError::RequirementFailed { + requirement: "assert-last-assigned-partition-id".into(), + detail: format!( + "last-partition-id is {}, expected {last_assigned_partition_id}", + metadata.last_partition_id + ), + }); + } } CommitRequirement::Unknown => { @@ -236,6 +244,18 @@ pub fn apply_updates( updates: &[CommitUpdate], branch: &str, ) -> Result<(), CommitError> { + // B16c: `-1` in `set-current-schema` / `set-default-spec` / + // `set-default-sort-order` means "the one added *by this commit*". The old + // code resolved it against `metadata.schemas.last()` etc., which for an + // existing table is never empty - so a `-1` sent *without* a preceding + // `add-schema` silently repointed the table at whatever happened to be last + // in the persisted vector (arbitrary if the vector is not in creation + // order), instead of hitting the error arm the message claims. Tracking + // what this commit actually added makes the sentinel mean what it says. + let mut last_added_schema_id: Option = None; + let mut last_added_spec_id: Option = None; + let mut last_added_sort_order_id: Option = None; + for update in updates { match update { CommitUpdate::AssignUuid { uuid } => { @@ -266,17 +286,33 @@ pub fn apply_updates( serde_json::from_value(schema.clone()).map_err(|e| CommitError::Invalid { detail: format!("add-schema: {e}"), })?; - metadata.last_column_id = metadata - .last_column_id - .max(new_schema.fields.iter().map(|f| f.id).max().unwrap_or(0)); + // Reject a duplicate id rather than pushing a second schema the + // rest of the metadata cannot tell apart (B16c). + if metadata + .schemas + .iter() + .any(|s| s.schema_id == new_schema.schema_id) + { + return Err(CommitError::Invalid { + detail: format!( + "add-schema: schema id {} already exists", + new_schema.schema_id + ), + }); + } + // `max_field_id` walks nested fields; taking the max over the + // top-level `fields` alone understates `last-column-id` for any + // schema containing a struct, list or map. + metadata.last_column_id = metadata.last_column_id.max(new_schema.max_field_id()); + last_added_schema_id = Some(new_schema.schema_id); metadata.schemas.push(new_schema); } CommitUpdate::SetCurrentSchema { schema_id } => { // -1 means "the schema added by this same commit". if *schema_id == -1 { - match metadata.schemas.last() { - Some(last) => metadata.current_schema_id = last.schema_id, + match last_added_schema_id { + Some(id) => metadata.current_schema_id = id, None => { return Err(CommitError::Invalid { detail: "set-current-schema: -1 with no schema in this commit" @@ -299,7 +335,7 @@ pub fn apply_updates( serde_json::from_value(snapshot.clone()).map_err(|e| CommitError::Invalid { detail: format!("add-snapshot: {e}"), })?; - add_snapshot(metadata, snapshot_obj, branch); + add_snapshot(metadata, snapshot_obj, branch)?; } CommitUpdate::SetSnapshotRef { @@ -371,18 +407,26 @@ pub fn apply_updates( serde_json::from_value(spec.clone()).map_err(|e| CommitError::Invalid { detail: format!("add-spec: {e}"), })?; + if metadata + .partition_specs + .iter() + .any(|s| s.spec_id == new_spec.spec_id) + { + return Err(CommitError::Invalid { + detail: format!("add-spec: spec id {} already exists", new_spec.spec_id), + }); + } + last_added_spec_id = Some(new_spec.spec_id); metadata.partition_specs.push(new_spec); + // Keep the required `last-partition-id` true (B12). + metadata.recompute_last_partition_id(); } CommitUpdate::SetDefaultSpec { spec_id } => { let target = if *spec_id == -1 { - metadata - .partition_specs - .last() - .map(|s| s.spec_id) - .ok_or_else(|| CommitError::Invalid { - detail: "set-default-spec: -1 with no spec in this commit".into(), - })? + last_added_spec_id.ok_or_else(|| CommitError::Invalid { + detail: "set-default-spec: -1 with no spec in this commit".into(), + })? } else { *spec_id }; @@ -401,19 +445,28 @@ pub fn apply_updates( detail: format!("add-sort-order: {e}"), } })?; + if metadata + .sort_orders + .iter() + .any(|o| o.order_id == new_order.order_id) + { + return Err(CommitError::Invalid { + detail: format!( + "add-sort-order: sort order id {} already exists", + new_order.order_id + ), + }); + } + last_added_sort_order_id = Some(new_order.order_id); metadata.sort_orders.push(new_order); } CommitUpdate::SetDefaultSortOrder { sort_order_id } => { let target = if *sort_order_id == -1 { - metadata - .sort_orders - .last() - .map(|o| o.order_id) - .ok_or_else(|| CommitError::Invalid { - detail: "set-default-sort-order: -1 with no sort order in this commit" - .into(), - })? + last_added_sort_order_id.ok_or_else(|| CommitError::Invalid { + detail: "set-default-sort-order: -1 with no sort order in this commit" + .into(), + })? } else { *sort_order_id }; @@ -466,7 +519,11 @@ pub fn apply_updates( } /// Append a snapshot and move the branch to it. -fn add_snapshot(metadata: &mut TableMetadata, snapshot: Snapshot, branch: &str) { +fn add_snapshot( + metadata: &mut TableMetadata, + snapshot: Snapshot, + branch: &str, +) -> Result<(), CommitError> { let snapshot_id = snapshot.snapshot_id; let timestamp_ms = if snapshot.timestamp_ms > 0 { snapshot.timestamp_ms @@ -475,10 +532,21 @@ fn add_snapshot(metadata: &mut TableMetadata, snapshot: Snapshot, branch: &str) }; // The sequence number is a monotonic counter, *not* a snapshot ID (A-3). - // Honour the client's value when it advances the counter; otherwise assign - // the next one. - let next_sequence = metadata.last_sequence_number + 1; - let sequence_number = if snapshot.sequence_number > metadata.last_sequence_number { + // + // B15: the old rule was "honour any client value greater than the current + // counter". A client could therefore submit `i64::MAX`, after which the next + // commit computed `last_sequence_number + 1` and overflowed - a panic in + // debug builds, a wrap in release, and corrupt commit ordering either way. + // The counter is the server's to advance: a client value is accepted only if + // it is exactly the next one, and anything else is assigned rather than + // honoured. + let next_sequence = metadata + .last_sequence_number + .checked_add(1) + .ok_or_else(|| CommitError::Invalid { + detail: "add-snapshot: sequence number counter is exhausted".into(), + })?; + let sequence_number = if snapshot.sequence_number == next_sequence { snapshot.sequence_number } else { next_sequence @@ -494,15 +562,22 @@ fn add_snapshot(metadata: &mut TableMetadata, snapshot: Snapshot, branch: &str) .push(snapshot); metadata.last_sequence_number = sequence_number; metadata.last_updated_ms = timestamp_ms; - metadata.current_snapshot_id = Some(snapshot_id); - metadata - .snapshot_log - .get_or_insert_with(Vec::new) - .push(SnapshotLogEntry { - timestamp_ms, - snapshot_id, - }); + // B16: `current_snapshot_id` and the `snapshot_log` describe *main*. Setting + // them for any branch meant a `dev`-branch commit changed what `main` + // readers resolve if the metadata document was ever shared across branches + // or exported. Same for fabricating a `main` ref pointing at the branch's + // snapshot: that silently published unreviewed work to main. + if branch == MAIN_REF { + metadata.current_snapshot_id = Some(snapshot_id); + metadata + .snapshot_log + .get_or_insert_with(Vec::new) + .push(SnapshotLogEntry { + timestamp_ms, + snapshot_id, + }); + } let refs = metadata.refs.get_or_insert_with(HashMap::new); refs.insert( @@ -515,18 +590,8 @@ fn add_snapshot(metadata: &mut TableMetadata, snapshot: Snapshot, branch: &str) max_ref_age_ms: None, }, ); - if branch != MAIN_REF { - // Keep `main` consistent with `current_snapshot_id` for readers that - // predate ref tracking. - refs.entry(MAIN_REF.to_string()) - .or_insert(SnapshotReference { - snapshot_id, - ref_type: "branch".to_string(), - min_snapshots_to_keep: None, - max_snapshot_age_ms: None, - max_ref_age_ms: None, - }); - } + + Ok(()) } #[cfg(test)] @@ -545,6 +610,7 @@ mod tests { last_column_id: 1, current_schema_id: 0, schemas: vec![Schema { + type_: "struct".to_string(), schema_id: 0, identifier_field_ids: None, fields: vec![NestedField { @@ -557,6 +623,7 @@ mod tests { }], current_partition_spec_id: 0, partition_specs: vec![], + last_partition_id: 999, default_sort_order_id: 0, sort_orders: vec![], properties: None, @@ -921,8 +988,12 @@ mod tests { assert_eq!(snapshots[1].sequence_number, 2); } + /// B15: the counter is the server's. A client value is honoured only when it + /// is exactly the next one; anything else is replaced by the next value + /// rather than letting the client jump the counter to, say, `i64::MAX` and + /// overflow the *following* commit. #[test] - fn a_client_supplied_sequence_number_may_advance_the_counter() { + fn a_client_supplied_sequence_number_cannot_jump_the_counter() { let mut metadata = base_metadata(); let mut snap = snapshot_json(1, None); snap["sequence-number"] = serde_json::json!(5); @@ -932,25 +1003,100 @@ mod tests { MAIN_REF, ) .unwrap(); - assert_eq!(metadata.last_sequence_number, 5); + assert_eq!(metadata.last_sequence_number, 1); + + // The exact next value is accepted. + let mut snap = snapshot_json(2, None); + snap["sequence-number"] = serde_json::json!(2); + apply_updates( + &mut metadata, + &[CommitUpdate::AddSnapshot { snapshot: snap }], + MAIN_REF, + ) + .unwrap(); + assert_eq!(metadata.last_sequence_number, 2); + } + + /// B15 regression: `i64::MAX` used to be honoured verbatim, so the next + /// commit's `last_sequence_number + 1` overflowed. + #[test] + fn an_absurd_client_sequence_number_does_not_overflow_the_next_commit() { + let mut metadata = base_metadata(); + let mut snap = snapshot_json(1, None); + snap["sequence-number"] = serde_json::json!(i64::MAX); + apply_updates( + &mut metadata, + &[CommitUpdate::AddSnapshot { snapshot: snap }], + MAIN_REF, + ) + .unwrap(); + assert_eq!(metadata.last_sequence_number, 1); + + // The follow-up commit is ordinary arithmetic, not an overflow. + apply_updates( + &mut metadata, + &[CommitUpdate::AddSnapshot { + snapshot: snapshot_json(2, None), + }], + MAIN_REF, + ) + .unwrap(); + assert_eq!(metadata.last_sequence_number, 2); } #[test] - fn adding_a_snapshot_records_the_snapshot_log_and_moves_the_branch() { + fn adding_a_snapshot_on_main_records_the_snapshot_log_and_moves_the_branch() { let mut metadata = base_metadata(); apply_updates( &mut metadata, &[CommitUpdate::AddSnapshot { snapshot: snapshot_json(99, None), }], - "feature", + MAIN_REF, ) .unwrap(); assert_eq!(metadata.current_snapshot_id, Some(99)); - assert_eq!(ref_snapshot_id(&metadata, "feature"), Some(99)); + assert_eq!(ref_snapshot_id(&metadata, MAIN_REF), Some(99)); assert_eq!(metadata.snapshot_log.as_ref().unwrap().len(), 1); } + /// B16: a commit to a feature branch must move *only* that branch. It used + /// to set `current_snapshot_id` for any branch and fabricate a `main` ref + /// pointing at the branch's snapshot, so a `dev` commit changed what `main` + /// readers resolve. + #[test] + fn committing_to_a_feature_branch_leaves_main_alone() { + let mut metadata = base_metadata(); + let main_before = ref_snapshot_id(&metadata, MAIN_REF); + + apply_updates( + &mut metadata, + &[CommitUpdate::AddSnapshot { + snapshot: snapshot_json(99, None), + }], + "feature", + ) + .unwrap(); + + assert_eq!(ref_snapshot_id(&metadata, "feature"), Some(99)); + assert_eq!( + ref_snapshot_id(&metadata, MAIN_REF), + main_before, + "a feature-branch commit must not move main" + ); + assert_eq!( + metadata.current_snapshot_id, main_before, + "current-snapshot-id describes main, not the committed branch" + ); + assert!( + metadata + .snapshot_log + .as_ref() + .is_none_or(|log| log.is_empty()), + "the snapshot log describes main's history" + ); + } + #[test] fn metadata_round_trips_through_json_with_refs() { let mut metadata = base_metadata(); diff --git a/pangolin/pangolin_api/src/iceberg/error.rs b/pangolin/pangolin_api/src/iceberg/error.rs index d05168d..9f1c1d7 100644 --- a/pangolin/pangolin_api/src/iceberg/error.rs +++ b/pangolin/pangolin_api/src/iceberg/error.rs @@ -80,6 +80,47 @@ pub fn forbidden(detail: &str) -> Response { iceberg_error(StatusCode::FORBIDDEN, "ForbiddenException", detail) } +/// `404` for a view that does not exist. +pub fn no_such_view(identifier: &str) -> Response { + iceberg_error( + StatusCode::NOT_FOUND, + "NoSuchViewException", + &format!("View does not exist: {identifier}"), + ) +} + +/// `400` for a malformed or unusable request. +pub fn bad_request(detail: &str) -> Response { + iceberg_error(StatusCode::BAD_REQUEST, "BadRequestException", detail) +} + +/// `409` for a table that already exists. +pub fn table_already_exists(identifier: &str) -> Response { + iceberg_error( + StatusCode::CONFLICT, + "AlreadyExistsException", + &format!("Table already exists: {identifier}"), + ) +} + +/// `409` for a namespace that already exists. +pub fn namespace_already_exists(namespace: &str) -> Response { + iceberg_error( + StatusCode::CONFLICT, + "AlreadyExistsException", + &format!("Namespace already exists: {namespace}"), + ) +} + +/// `409` for a namespace that still has children. +pub fn namespace_not_empty(namespace: &str) -> Response { + iceberg_error( + StatusCode::CONFLICT, + "NamespaceNotEmptyException", + &format!("Namespace is not empty: {namespace}"), + ) +} + /// `500`, with the underlying cause logged rather than returned. pub fn internal(context: &str) -> Response { iceberg_error( diff --git a/pangolin/pangolin_api/src/iceberg/mod.rs b/pangolin/pangolin_api/src/iceberg/mod.rs index 1b93310..db57d9a 100644 --- a/pangolin/pangolin_api/src/iceberg/mod.rs +++ b/pangolin/pangolin_api/src/iceberg/mod.rs @@ -18,7 +18,10 @@ pub mod tables; pub mod types; // Re-export types for convenience -pub use error::iceberg_error; +pub use error::{ + bad_request, forbidden, iceberg_error, internal, namespace_already_exists, namespace_not_empty, + no_such_namespace, no_such_table, no_such_view, table_already_exists, +}; pub use types::*; pub type AppState = std::sync::Arc; diff --git a/pangolin/pangolin_api/src/iceberg/namespaces.rs b/pangolin/pangolin_api/src/iceberg/namespaces.rs index d2dc88f..770af8e 100644 --- a/pangolin/pangolin_api/src/iceberg/namespaces.rs +++ b/pangolin/pangolin_api/src/iceberg/namespaces.rs @@ -1,5 +1,8 @@ use super::types::*; -use super::{check_and_forward_if_federated, AppState}; +use super::{ + check_and_forward_if_federated, forbidden, internal, namespace_already_exists, + no_such_namespace, AppState, +}; use crate::auth::TenantId; use crate::authz::check_permission; use axum::{ @@ -37,7 +40,7 @@ pub async fn list_namespaces( Extension(session): Extension, Path(prefix): Path, Query(params): Query, - Query(pagination): Query, + Query(page): Query, ) -> impl IntoResponse { let tenant_id = tenant.0; let catalog_name = prefix.clone(); @@ -67,9 +70,10 @@ pub async fn list_namespaces( // Local catalog handling let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { Ok(Some(c)) => c, - Ok(None) => return (StatusCode::NOT_FOUND, "Catalog not found").into_response(), - Err(_) => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response() + Ok(None) => return no_such_namespace(&catalog_name), + Err(e) => { + tracing::error!(error = %e, "list_namespaces: failed to load catalog"); + return internal("Failed to load catalog"); } }; @@ -79,31 +83,193 @@ pub async fn list_namespaces( }; match check_permission(&store, &session, &Action::List, &scope).await { Ok(true) => (), - Ok(false) => return (StatusCode::FORBIDDEN, "Forbidden").into_response(), + Ok(false) => return forbidden("Forbidden"), Err(e) => { - return ( - StatusCode::INTERNAL_SERVER_ERROR, - format!("Permission check failed: {}", e), - ) - .into_response() + tracing::error!(error = %e, "list_namespaces: permission check failed"); + return internal("Permission check failed"); } } + let (offset, limit) = page.resolve(); + let pagination = PaginationParams { + limit: Some(limit as usize), + offset: Some(offset as usize), + }; + match store .list_namespaces(tenant_id, &catalog_name, params.parent, Some(pagination)) .await { Ok(namespaces) => { + let returned = namespaces.len(); let ns_list: Vec> = namespaces.into_iter().map(|n| n.name).collect(); ( StatusCode::OK, Json(ListNamespacesResponse { namespaces: ns_list, + next_page_token: next_page_token(returned, offset, limit), }), ) .into_response() } - Err(_) => (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response(), + Err(e) => { + tracing::error!(error = %e, "list_namespaces: failed to list namespaces"); + internal("Failed to list namespaces") + } + } +} + +/// Load a namespace's metadata (`loadNamespaceMetadata`). +/// +/// Part of completing the Iceberg REST surface: this endpoint was on the +/// README's "not implemented" list, so clients could create a namespace and set +/// its properties but never read them back. +#[utoipa::path( + get, + path = "/v1/{prefix}/namespaces/{namespace}", + tag = "Iceberg REST", + params( + ("prefix" = String, Path, description = "Catalog name"), + ("namespace" = String, Path, description = "Namespace name") + ), + responses( + (status = 200, description = "Namespace metadata", body = CreateNamespaceResponse), + (status = 403, description = "Forbidden"), + (status = 404, description = "Namespace not found"), + (status = 500, description = "Internal server error") + ), + security(("bearer_auth" = [])) +)] +pub async fn load_namespace_metadata( + State(store): State, + Extension(tenant): Extension, + Extension(session): Extension, + Path((prefix, namespace)): Path<(String, String)>, +) -> impl IntoResponse { + let tenant_id = tenant.0; + let catalog_name = prefix; + + let path = format!("/namespaces/{}", namespace); + if let Some(response) = check_and_forward_if_federated( + &store, + tenant_id, + &catalog_name, + Method::GET, + &path, + None, + HeaderMap::new(), + ) + .await + { + return response; + } + + let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { + Ok(Some(c)) => c, + Ok(None) => return no_such_namespace(&catalog_name), + Err(e) => { + tracing::error!(error = %e, "load_namespace_metadata: failed to load catalog"); + return internal("Failed to load catalog"); + } + }; + + let (namespace_parts, _branch) = parse_namespace(&namespace); + + let scope = PermissionScope::Namespace { + catalog_id: catalog.id, + namespace: namespace_parts.join("."), + }; + match check_permission(&store, &session, &Action::Read, &scope).await { + Ok(true) => (), + Ok(false) => return forbidden("Forbidden"), + Err(e) => { + tracing::error!(error = %e, "load_namespace_metadata: permission check failed"); + return internal("Permission check failed"); + } + } + + match store + .get_namespace(tenant_id, &catalog_name, namespace_parts.clone()) + .await + { + Ok(Some(ns)) => ( + StatusCode::OK, + Json(CreateNamespaceResponse { + namespace: ns.name, + properties: ns.properties, + }), + ) + .into_response(), + Ok(None) => no_such_namespace(&namespace_parts.join(".")), + Err(e) => { + tracing::error!(error = %e, "load_namespace_metadata: failed to load namespace"); + internal("Failed to load namespace") + } + } +} + +/// Check whether a namespace exists (`namespaceExists`). +/// +/// `HEAD` with an empty body, per the spec. Also on the README's +/// "not implemented" list. +#[utoipa::path( + head, + path = "/v1/{prefix}/namespaces/{namespace}", + tag = "Iceberg REST", + params( + ("prefix" = String, Path, description = "Catalog name"), + ("namespace" = String, Path, description = "Namespace name") + ), + responses( + (status = 204, description = "Namespace exists"), + (status = 403, description = "Forbidden"), + (status = 404, description = "Namespace not found") + ), + security(("bearer_auth" = [])) +)] +pub async fn namespace_exists( + State(store): State, + Extension(tenant): Extension, + Extension(session): Extension, + Path((prefix, namespace)): Path<(String, String)>, +) -> impl IntoResponse { + let tenant_id = tenant.0; + let catalog_name = prefix; + + let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { + Ok(Some(c)) => c, + Ok(None) => return StatusCode::NOT_FOUND.into_response(), + Err(e) => { + tracing::error!(error = %e, "namespace_exists: failed to load catalog"); + return StatusCode::INTERNAL_SERVER_ERROR.into_response(); + } + }; + + let (namespace_parts, _branch) = parse_namespace(&namespace); + + let scope = PermissionScope::Namespace { + catalog_id: catalog.id, + namespace: namespace_parts.join("."), + }; + match check_permission(&store, &session, &Action::Read, &scope).await { + Ok(true) => (), + Ok(false) => return StatusCode::FORBIDDEN.into_response(), + Err(e) => { + tracing::error!(error = %e, "namespace_exists: permission check failed"); + return StatusCode::INTERNAL_SERVER_ERROR.into_response(); + } + } + + match store + .get_namespace(tenant_id, &catalog_name, namespace_parts) + .await + { + Ok(Some(_)) => StatusCode::NO_CONTENT.into_response(), + Ok(None) => StatusCode::NOT_FOUND.into_response(), + Err(e) => { + tracing::error!(error = %e, "namespace_exists: failed to load namespace"); + StatusCode::INTERNAL_SERVER_ERROR.into_response() + } } } @@ -140,12 +306,34 @@ pub async fn create_namespace( catalog_name ); + // B16k: federated forwarding was missing here (and on delete and the + // namespace tree) although `list_namespaces` had it. On a `Federated` + // catalog, creating a namespace built a *local shadow* and returned 200 + // while `GET` listed the remote - the two views diverged permanently, and + // delete reported success for a namespace still present upstream. + let path = "/namespaces".to_string(); + let body_bytes = serde_json::to_vec(&payload).ok().map(Bytes::from); + if let Some(response) = check_and_forward_if_federated( + &store, + tenant_id, + &catalog_name, + Method::POST, + &path, + body_bytes, + HeaderMap::new(), + ) + .await + { + return response; + } + // Resolve catalog ID let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { Ok(Some(c)) => c, - Ok(None) => return (StatusCode::NOT_FOUND, "Catalog not found").into_response(), - Err(_) => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response() + Ok(None) => return no_such_namespace(&catalog_name), + Err(e) => { + tracing::error!(error = %e, "create_namespace: failed to load catalog"); + return internal("Failed to load catalog"); } }; @@ -155,13 +343,10 @@ pub async fn create_namespace( }; match check_permission(&store, &session, &Action::Create, &scope).await { Ok(true) => (), - Ok(false) => return (StatusCode::FORBIDDEN, "Forbidden").into_response(), + Ok(false) => return forbidden("Forbidden"), Err(e) => { - return ( - StatusCode::INTERNAL_SERVER_ERROR, - format!("Permission check failed: {}", e), - ) - .into_response() + tracing::error!(error = %e, "create_namespace: permission check failed"); + return internal("Permission check failed"); } } @@ -170,6 +355,21 @@ pub async fn create_namespace( properties: payload.properties.unwrap_or_default(), }; + // Report a conflict rather than silently overwriting an existing namespace's + // properties, which is what the create path did on backends whose insert is + // an upsert. + match store + .get_namespace(tenant_id, &catalog_name, ns.name.clone()) + .await + { + Ok(Some(_)) => return namespace_already_exists(&ns.name.join(".")), + Ok(None) => {} + Err(e) => { + tracing::error!(error = %e, "create_namespace: existence check failed"); + return internal("Failed to check namespace"); + } + } + match store .create_namespace(tenant_id, &catalog_name, ns.clone()) .await @@ -200,7 +400,10 @@ pub async fn create_namespace( ) .into_response() } - Err(_) => (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response(), + Err(e) => { + tracing::error!(error = %e, "create_namespace: failed to create namespace"); + internal("Failed to create namespace") + } } } @@ -230,16 +433,33 @@ pub async fn delete_namespace( let tenant_id = tenant.0; let catalog_name = prefix; + // Federated forwarding (B16k) - see `create_namespace`. + let path = format!("/namespaces/{}", namespace); + if let Some(response) = check_and_forward_if_federated( + &store, + tenant_id, + &catalog_name, + Method::DELETE, + &path, + None, + HeaderMap::new(), + ) + .await + { + return response; + } + // Resolve catalog ID let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { Ok(Some(c)) => c, - Ok(None) => return (StatusCode::NOT_FOUND, "Catalog not found").into_response(), - Err(_) => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response() + Ok(None) => return no_such_namespace(&catalog_name), + Err(e) => { + tracing::error!(error = %e, "delete_namespace: failed to load catalog"); + return internal("Failed to load catalog"); } }; - let namespace_parts: Vec = namespace.split('\x1F').map(|s| s.to_string()).collect(); + let (namespace_parts, _branch) = parse_namespace(&namespace); // Check Permissions let scope = PermissionScope::Namespace { @@ -249,13 +469,10 @@ pub async fn delete_namespace( match check_permission(&store, &session, &Action::Delete, &scope).await { Ok(true) => (), - Ok(false) => return (StatusCode::FORBIDDEN, "Forbidden").into_response(), + Ok(false) => return forbidden("Forbidden"), Err(e) => { - return ( - StatusCode::INTERNAL_SERVER_ERROR, - format!("Permission check failed: {}", e), - ) - .into_response() + tracing::error!(error = %e, "delete_namespace: permission check failed"); + return internal("Permission check failed"); } } @@ -282,11 +499,25 @@ pub async fn delete_namespace( StatusCode::NO_CONTENT.into_response() } - Err(_) => (StatusCode::NOT_FOUND, "Namespace not found").into_response(), + Err(e) => { + tracing::debug!(error = %e, "delete_namespace: namespace not deleted"); + no_such_namespace(&namespace_parts.join(".")) + } } } -/// Update namespace properties +/// Update a namespace's properties. +/// +/// Two fixes here: +/// +/// * **B0d** - the handler bound `Extension(_session)` (deliberately discarding +/// it) and never called `check_permission`, and never resolved the catalog at +/// all. Any tenant member could rewrite any namespace's properties, including +/// `location`, which later table creation derives paths from. +/// * **B16h** - `removals` were silently ignored. A request carrying removals +/// got `200 OK` with `removed: []` / `missing: []` while nothing was removed: +/// exactly the "silent success" failure class 0.6.0 set out to eliminate. The +/// three response lists are now reported honestly. #[utoipa::path( post, path = "/v1/{prefix}/namespaces/{namespace}/properties", @@ -307,7 +538,7 @@ pub async fn delete_namespace( pub async fn update_namespace_properties( State(store): State, Extension(tenant): Extension, - Extension(_session): Extension, + Extension(session): Extension, Path((prefix, namespace)): Path<(String, String)>, Json(payload): Json, ) -> impl IntoResponse { @@ -331,26 +562,44 @@ pub async fn update_namespace_properties( { return response; } - let namespace_parts: Vec = namespace.split('\x1F').map(|s| s.to_string()).collect(); - - // For MVP, we only support updates. Removals are ignored or TODO. - if let Some(updates) = payload.updates { - match store - .update_namespace_properties(tenant_id, &catalog_name, namespace_parts, updates.clone()) - .await - { - Ok(_) => { - let response = UpdateNamespacePropertiesResponse { - updated: updates.keys().cloned().collect(), - removed: vec![], - missing: vec![], - }; - (StatusCode::OK, Json(response)).into_response() - } - Err(_) => (StatusCode::NOT_FOUND, "Namespace not found").into_response(), + + let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { + Ok(Some(c)) => c, + Ok(None) => return no_such_namespace(&catalog_name), + Err(e) => { + tracing::error!(error = %e, "update_namespace_properties: failed to load catalog"); + return internal("Failed to load catalog"); + } + }; + + let (namespace_parts, _branch) = parse_namespace(&namespace); + + let scope = PermissionScope::Namespace { + catalog_id: catalog.id, + namespace: namespace_parts.join("."), + }; + match check_permission(&store, &session, &Action::Write, &scope).await { + Ok(true) => (), + Ok(false) => return forbidden("Forbidden"), + Err(e) => { + tracing::error!(error = %e, "update_namespace_properties: permission check failed"); + return internal("Permission check failed"); } - } else { - ( + } + + let updates = payload.updates.unwrap_or_default(); + let removals = payload.removals.unwrap_or_default(); + + // The spec rejects a key that appears in both lists rather than picking a + // winner. + if let Some(conflict) = removals.iter().find(|k| updates.contains_key(*k)) { + return super::bad_request(&format!( + "Property {conflict} appears in both updates and removals" + )); + } + + if updates.is_empty() && removals.is_empty() { + return ( StatusCode::OK, Json(UpdateNamespacePropertiesResponse { updated: vec![], @@ -358,7 +607,59 @@ pub async fn update_namespace_properties( missing: vec![], }), ) - .into_response() + .into_response(); + } + + // Read-modify-write: removals cannot be expressed by the merging store + // method, so the resulting map is computed here and written wholesale. + let existing = match store + .get_namespace(tenant_id, &catalog_name, namespace_parts.clone()) + .await + { + Ok(Some(ns)) => ns.properties, + Ok(None) => return no_such_namespace(&namespace_parts.join(".")), + Err(e) => { + tracing::error!(error = %e, "update_namespace_properties: failed to load namespace"); + return internal("Failed to load namespace"); + } + }; + + let mut properties = existing; + let mut removed = Vec::new(); + let mut missing = Vec::new(); + for key in &removals { + if properties.remove(key).is_some() { + removed.push(key.clone()); + } else { + missing.push(key.clone()); + } + } + + let updated: Vec = updates.keys().cloned().collect(); + properties.extend(updates); + + match store + .replace_namespace_properties( + tenant_id, + &catalog_name, + namespace_parts.clone(), + properties, + ) + .await + { + Ok(_) => ( + StatusCode::OK, + Json(UpdateNamespacePropertiesResponse { + updated, + removed, + missing, + }), + ) + .into_response(), + Err(e) => { + tracing::error!(error = %e, "update_namespace_properties: failed to write properties"); + no_such_namespace(&namespace_parts.join(".")) + } } } @@ -387,12 +688,29 @@ pub async fn list_namespaces_tree( let tenant_id = tenant.0; let catalog_name = prefix.clone(); + // Federated forwarding (B16k): without it the tree renders the local shadow + // while every other view shows the remote. + if let Some(response) = check_and_forward_if_federated( + &store, + tenant_id, + &catalog_name, + Method::GET, + "/namespaces", + None, + HeaderMap::new(), + ) + .await + { + return response; + } + // Resolve catalog ID let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { Ok(Some(c)) => c, - Ok(None) => return (StatusCode::NOT_FOUND, "Catalog not found").into_response(), - Err(_) => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response() + Ok(None) => return no_such_namespace(&catalog_name), + Err(e) => { + tracing::error!(error = %e, "list_namespaces_tree: failed to load catalog"); + return internal("Failed to load catalog"); } }; @@ -402,13 +720,10 @@ pub async fn list_namespaces_tree( }; match check_permission(&store, &session, &Action::List, &scope).await { Ok(true) => (), - Ok(false) => return (StatusCode::FORBIDDEN, "Forbidden").into_response(), + Ok(false) => return forbidden("Forbidden"), Err(e) => { - return ( - StatusCode::INTERNAL_SERVER_ERROR, - format!("Permission check failed: {}", e), - ) - .into_response() + tracing::error!(error = %e, "list_namespaces_tree: permission check failed"); + return internal("Permission check failed"); } } @@ -455,10 +770,9 @@ pub async fn list_namespaces_tree( ) .into_response() } - Err(e) => ( - StatusCode::INTERNAL_SERVER_ERROR, - format!("Failed to list namespaces: {}", e), - ) - .into_response(), + Err(e) => { + tracing::error!(error = %e, "list_namespaces_tree: failed to list namespaces"); + internal("Failed to list namespaces") + } } } diff --git a/pangolin/pangolin_api/src/iceberg/oauth.rs b/pangolin/pangolin_api/src/iceberg/oauth.rs index 9ff80c2..32e1c42 100644 --- a/pangolin/pangolin_api/src/iceberg/oauth.rs +++ b/pangolin/pangolin_api/src/iceberg/oauth.rs @@ -86,9 +86,15 @@ pub async fn handle_oauth_token( .map_err(ApiError::InternalError)? .ok_or_else(|| ApiError::unauthorized("Invalid client_id"))?; - // 4. Verify Active Status - if !service_user.active { - return Err(ApiError::unauthorized("Client is inactive")); + // 4. Verify Active Status *and* expiry. + // + // B0g: this checked only `active`, not `is_valid()` (= `active && + // !is_expired()`). The API-key path in `auth_middleware` uses `is_valid()` + // correctly, so an *expired* service user was refused there but could still + // exchange `client_credentials` here for a fresh 1-hour JWT - fully + // bypassing key expiry, and renewably. + if !service_user.is_valid() { + return Err(ApiError::unauthorized("Client is inactive or expired")); } // 5. Verify Secret (API Key) diff --git a/pangolin/pangolin_api/src/iceberg/tables.rs b/pangolin/pangolin_api/src/iceberg/tables.rs index 4c1bfdd..83ca3d1 100644 --- a/pangolin/pangolin_api/src/iceberg/tables.rs +++ b/pangolin/pangolin_api/src/iceberg/tables.rs @@ -1,5 +1,8 @@ use super::types::*; -use super::{check_and_forward_if_federated, commit, iceberg_error, AppState}; +use super::{ + bad_request, check_and_forward_if_federated, commit, forbidden, iceberg_error, internal, + no_such_namespace, no_such_table, table_already_exists, AppState, +}; use crate::auth::TenantId; use crate::authz::check_permission; use axum::{ @@ -11,7 +14,7 @@ use axum::{ use bytes::Bytes; use chrono::Utc; use pangolin_core::iceberg_metadata::{ - NestedField, PartitionSpec, Schema, SortOrder, TableMetadata, Type, + MetadataLogEntry, PartitionSpec, Schema, SortOrder, TableMetadata, }; use pangolin_core::model::{Asset, AssetType}; use pangolin_core::permission::{Action, PermissionScope}; @@ -21,6 +24,11 @@ use std::collections::HashMap; use std::sync::Arc; use uuid::Uuid; +/// How many previous metadata files to keep in `metadata-log` when the table +/// does not set `write.metadata.previous-versions-max`. Matches the Iceberg +/// default. +const DEFAULT_PREVIOUS_VERSIONS_MAX: usize = 100; + /// List tables in a namespace #[utoipa::path( get, @@ -43,7 +51,7 @@ pub async fn list_tables( Extension(tenant): Extension, Extension(session): Extension, Path((prefix, namespace)): Path<(String, String)>, - Query(pagination): Query, + Query(page): Query, ) -> impl IntoResponse { let tenant_id = tenant.0; let catalog_name = prefix.clone(); @@ -66,32 +74,34 @@ pub async fn list_tables( let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { Ok(Some(c)) => c, - Ok(None) => return (StatusCode::NOT_FOUND, "Catalog not found").into_response(), - Err(_) => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response() + Ok(None) => return no_such_namespace(&catalog_name), + Err(e) => { + tracing::error!(error = %e, "list_tables: failed to load catalog"); + return internal("Failed to load catalog"); } }; - let (ns_name, branch) = parse_table_identifier(&namespace); + let (ns_vec, branch) = parse_namespace(&namespace); // Check Permissions let scope = PermissionScope::Namespace { catalog_id: catalog.id, - namespace: ns_name.clone(), + namespace: ns_vec.join("."), }; match check_permission(&store, &session, &Action::List, &scope).await { Ok(true) => (), - Ok(false) => return (StatusCode::FORBIDDEN, "Forbidden").into_response(), + Ok(false) => return forbidden("Forbidden"), Err(e) => { - return ( - StatusCode::INTERNAL_SERVER_ERROR, - format!("Permission check failed: {}", e), - ) - .into_response() + tracing::error!(error = %e, "list_tables: permission check failed"); + return internal("Permission check failed"); } } - let ns_vec = vec![ns_name]; + let (offset, limit) = page.resolve(); + let pagination = PaginationParams { + limit: Some(limit as usize), + offset: Some(offset as usize), + }; match store .list_assets( @@ -104,6 +114,11 @@ pub async fn list_tables( .await { Ok(assets) => { + // The page token has to be computed from the number of rows the + // store returned, before the asset-type filter narrows it - the + // store is what applied `limit`, so a full page of rows means there + // may be more even if none of them survive the filter. + let returned = assets.len(); let identifiers: Vec = assets .into_iter() .filter(|a| a.kind == AssetType::IcebergTable) @@ -112,9 +127,19 @@ pub async fn list_tables( name: a.name, }) .collect(); - (StatusCode::OK, Json(ListTablesResponse { identifiers })).into_response() + ( + StatusCode::OK, + Json(ListTablesResponse { + identifiers, + next_page_token: next_page_token(returned, offset, limit), + }), + ) + .into_response() + } + Err(e) => { + tracing::error!(error = %e, "list_tables: failed to list assets"); + internal("Failed to list tables") } - Err(_) => (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response(), } } @@ -126,6 +151,13 @@ pub struct MaintenanceRequest { } /// Perform maintenance on a table +/// +/// Two bugs fixed here (B0f): +/// 1. the catalog from the path was discarded and the literal `"default"` was +/// passed to `expire_snapshots`/`remove_orphan_files`, so destructive +/// maintenance ran against the wrong catalog entirely; +/// 2. there was no session and no permission check, so any tenant member could +/// trigger snapshot expiry and orphan-file deletion on any table. #[utoipa::path( post, path = "/api/v1/catalogs/{prefix}/namespaces/{namespace}/tables/{table}/maintenance", @@ -139,6 +171,8 @@ pub struct MaintenanceRequest { responses( (status = 200, description = "Maintenance accepted", body = serde_json::Value), (status = 400, description = "Bad request"), + (status = 403, description = "Forbidden"), + (status = 404, description = "Catalog or table not found"), (status = 500, description = "Internal server error") ), security(("bearer_auth" = [])) @@ -146,24 +180,67 @@ pub struct MaintenanceRequest { pub async fn perform_maintenance( State(store): State>, Extension(tenant_id): Extension, - Path((_prefix, namespace, table)): Path<(String, String, String)>, + Extension(session): Extension, + Path((prefix, namespace, table)): Path<(String, String, String)>, Json(payload): Json, ) -> Result, StatusCode> { - let namespace_parts: Vec = namespace.split('\x1F').map(|s| s.to_string()).collect(); - // Parse table@branch - let (table_name, branch_name) = if let Some((t, b)) = table.split_once('@') { - (t.to_string(), Some(b.to_string())) - } else { - (table.to_string(), None) + let tenant = tenant_id.0; + let catalog_name = prefix; + + let (namespace_parts, branch_from_ns) = parse_namespace(&namespace); + let (table_name, branch_from_table) = parse_table_identifier(&table); + let branch_name = branch_from_table.or(branch_from_ns); + + let catalog = match store.get_catalog(tenant, catalog_name.clone()).await { + Ok(Some(c)) => c, + Ok(None) => return Err(StatusCode::NOT_FOUND), + Err(e) => { + tracing::error!(error = %e, "maintenance: failed to load catalog"); + return Err(StatusCode::INTERNAL_SERVER_ERROR); + } }; + let asset = match store + .get_asset( + tenant, + &catalog_name, + branch_name.clone(), + namespace_parts.clone(), + table_name.clone(), + ) + .await + { + Ok(Some(a)) => a, + Ok(None) => return Err(StatusCode::NOT_FOUND), + Err(e) => { + tracing::error!(error = %e, "maintenance: failed to load asset"); + return Err(StatusCode::INTERNAL_SERVER_ERROR); + } + }; + + // Snapshot expiry and orphan-file removal both destroy data, so they need + // Delete, not merely Write. + let scope = PermissionScope::Asset { + catalog_id: catalog.id, + namespace: namespace_parts.join("."), + asset_id: asset.id, + }; + match check_permission(&store, &session, &Action::Delete, &scope).await { + Ok(true) => (), + Ok(false) => return Err(StatusCode::FORBIDDEN), + Err(e) => { + tracing::error!(error = %e, "maintenance: permission check failed"); + return Err(StatusCode::INTERNAL_SERVER_ERROR); + } + } + match payload.job_type.as_str() { "expire_snapshots" => { let retention = payload.retention_ms.unwrap_or(86400000); // Default 1 day store .expire_snapshots( - tenant_id.0, - "default", + tenant, + &catalog_name, branch_name, namespace_parts, table_name, @@ -179,8 +256,8 @@ pub async fn perform_maintenance( let older_than = payload.older_than_ms.unwrap_or(86400000); // Default 1 day store .remove_orphan_files( - tenant_id.0, - "default", + tenant, + &catalog_name, branch_name, namespace_parts, table_name, @@ -256,16 +333,18 @@ pub async fn create_table( let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { Ok(Some(c)) => c, - Ok(None) => return (StatusCode::NOT_FOUND, "Catalog not found").into_response(), - Err(_) => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response() + Ok(None) => return no_such_namespace(&catalog_name), + Err(e) => { + tracing::error!(error = %e, "create_table: failed to load catalog"); + return internal("Failed to load catalog"); } }; let (tbl_name, branch_from_name) = parse_table_identifier(&payload.name); - let (ns_name, branch_from_ns) = parse_table_identifier(&namespace); + let (ns_vec, branch_from_ns) = parse_namespace(&namespace); let branch_from_query = params.get("branch").cloned(); let branch = branch_from_name.or(branch_from_ns).or(branch_from_query); + let ns_name = ns_vec.join("."); let scope = PermissionScope::Namespace { catalog_id: catalog.id, @@ -273,18 +352,13 @@ pub async fn create_table( }; match check_permission(&store, &session, &Action::Create, &scope).await { Ok(true) => (), - Ok(false) => return (StatusCode::FORBIDDEN, "Forbidden").into_response(), + Ok(false) => return forbidden("Forbidden"), Err(e) => { - return ( - StatusCode::INTERNAL_SERVER_ERROR, - format!("Permission check failed: {}", e), - ) - .into_response() + tracing::error!(error = %e, "create_table: permission check failed"); + return internal("Permission check failed"); } } - let ns_vec = vec![ns_name.clone()]; - let table_uuid = Uuid::new_v4(); let location = if let Some(loc) = payload.location { loc @@ -328,66 +402,57 @@ pub async fn create_table( ) }; - let schema_fields = if let Some(schema_value) = &payload.schema { - if let Some(fields) = schema_value.get("fields").and_then(|f| f.as_array()) { - fields - .iter() - .filter_map(|field| { - let id = field.get("id")?.as_i64()? as i32; - let name = field.get("name")?.as_str()?.to_string(); - let required = false; - let field_type_str = field.get("type")?.as_str()?; - - let field_type = match field_type_str { - "int" | "integer" => Type::Primitive("long".to_string()), - "long" => Type::Primitive("long".to_string()), - "string" => Type::Primitive("string".to_string()), - "boolean" => Type::Primitive("boolean".to_string()), - "float" => Type::Primitive("float".to_string()), - "double" => Type::Primitive("double".to_string()), - "date" => Type::Primitive("date".to_string()), - "time" => Type::Primitive("time".to_string()), - "timestamp" => Type::Primitive("timestamp".to_string()), - "timestamptz" => Type::Primitive("timestamptz".to_string()), - "binary" => Type::Primitive("binary".to_string()), - "uuid" => Type::Primitive("uuid".to_string()), - _ => Type::Primitive(field_type_str.to_string()), - }; - - Some(NestedField { - id, - name, - required, - field_type, - doc: None, - }) - }) - .collect() - } else { - vec![] - } - } else { - vec![] + // B16f: the schema used to be hand-parsed field by field, which silently + // lost data three ways - `required` was hardcoded to `false` so every column + // became optional, `int` was widened to `long`, and `field.get("type")? + // .as_str()?` inside a `filter_map` returned `None` for any struct/list/map/ + // decimal/fixed column, so those columns were *dropped* and `last_column_id` + // was computed from the survivors. The table was created `200 OK` with a + // schema missing columns. + // + // Deserializing straight into the core `Schema` (as the commit path already + // does) keeps nullability and complex types, and a malformed schema is now a + // 400 rather than a quietly mangled table. + let schema = match &payload.schema { + Some(schema_value) => match serde_json::from_value::(schema_value.clone()) { + Ok(mut s) => { + s.schema_id = 0; + if s.identifier_field_ids.is_none() { + s.identifier_field_ids = Some(vec![]); + } + s + } + Err(e) => { + return bad_request(&format!("Invalid schema: {}", e)); + } + }, + None => Schema { + type_: Schema::STRUCT.to_string(), + schema_id: 0, + identifier_field_ids: Some(vec![]), + fields: vec![], + }, }; + let last_column_id = schema.max_field_id(); + let metadata = TableMetadata { format_version: 2, table_uuid, location: location.clone(), last_sequence_number: 0, last_updated_ms: Utc::now().timestamp_millis(), - last_column_id: schema_fields.iter().map(|f| f.id).max().unwrap_or(0), - schemas: vec![Schema { - schema_id: 0, - identifier_field_ids: Some(vec![]), - fields: schema_fields, - }], + last_column_id, + schemas: vec![schema], current_schema_id: 0, current_partition_spec_id: 0, partition_specs: vec![PartitionSpec { spec_id: 0, fields: vec![], }], + // Required by the v2 spec; an empty unpartitioned spec assigns nothing, + // so the highest assigned partition id is the "unpartitioned" sentinel. + last_partition_id: pangolin_core::iceberg_metadata::PARTITION_FIELD_ID_START - 1, default_sort_order_id: 0, sort_orders: vec![SortOrder { order_id: 0, @@ -401,7 +466,13 @@ pub async fn create_table( refs: None, }; - let metadata_json = serde_json::to_string(&metadata).unwrap(); + let metadata_json = match serde_json::to_string(&metadata) { + Ok(json) => json, + Err(e) => { + tracing::error!(error = %e, "create_table: failed to serialize metadata"); + return internal("Failed to serialize table metadata"); + } + }; let metadata_location = format!( "{}/metadata/00000-{}.metadata.json", location, @@ -423,23 +494,30 @@ pub async fn create_table( }, }; + // B16g: write the metadata file *first*, then register the asset. The old + // order registered the asset and only then wrote the file, so a failed write + // left a permanently broken table - registered, pointing at a file that does + // not exist, with `load_table` 500ing and `update_table` 404ing and no repair + // path. This is also the order the commit path already uses. + if let Err(e) = store + .write_file(&metadata_location, metadata_json.into_bytes()) + .await + { + tracing::error!("Failed to write metadata file: {}", e); + return internal("Failed to write metadata"); + } + match store - .create_asset(tenant_id, &catalog_name, branch, ns_vec, asset.clone()) + .create_asset( + tenant_id, + &catalog_name, + branch, + ns_vec.clone(), + asset.clone(), + ) .await { Ok(_) => { - if let Err(e) = store - .write_file(&metadata_location, metadata_json.into_bytes()) - .await - { - tracing::error!("Failed to write metadata file: {}", e); - return ( - StatusCode::INTERNAL_SERVER_ERROR, - "Failed to write metadata", - ) - .into_response(); - } - let _ = store .log_audit_event( tenant_id, @@ -493,7 +571,12 @@ pub async fn create_table( ( StatusCode::OK, Json(TableResponse::with_credentials( - Some(location.clone()), + // B16e: this returned `location` - the table *directory* - + // where the spec (and `load_table`, correctly) return the + // metadata *file*. A client that keeps the returned `Table` + // (PyIceberg does) ended up with a `metadata_location` it + // could neither read nor refresh from. + Some(metadata_location.clone()), metadata, credentials, Some(table_uuid), @@ -501,7 +584,20 @@ pub async fn create_table( ) .into_response() } - Err(_) => (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response(), + Err(e) => { + tracing::error!(error = %e, "create_table: failed to register asset"); + // The metadata file was written before registration (B16g); with the + // asset unregistered it is unreferenced, so clean it up rather than + // leaving an orphan behind. + if let Err(cleanup) = store.delete_file(&metadata_location).await { + tracing::warn!( + error = %cleanup, + location = %metadata_location, + "could not remove the orphaned metadata file after a failed create_asset" + ); + } + internal("Failed to create table") + } } } @@ -560,18 +656,19 @@ pub async fn load_table( let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { Ok(Some(c)) => c, - Ok(None) => return (StatusCode::NOT_FOUND, "Catalog not found").into_response(), - Err(_) => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response() + Ok(None) => return no_such_namespace(&catalog_name), + Err(e) => { + tracing::error!(error = %e, "load_table: failed to load catalog"); + return internal("Failed to load catalog"); } }; + // B16a: shared namespace parsing, so a nested namespace resolves the same + // way here as it does on the commit path. let (tbl_name, branch_from_name) = parse_table_identifier(&table); - let (ns_name, branch_from_ns) = parse_table_identifier(&namespace); + let (ns_vec, branch_from_ns) = parse_namespace(&namespace); let branch = branch_from_name.or(branch_from_ns); - let ns_vec = vec![ns_name]; - let asset = match store .get_asset( tenant_id, @@ -583,9 +680,10 @@ pub async fn load_table( .await { Ok(Some(a)) => a, - Ok(None) => return (StatusCode::NOT_FOUND, "Table not found").into_response(), - Err(_) => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response() + Ok(None) => return no_such_table(&format!("{}.{}", ns_vec.join("."), tbl_name)), + Err(e) => { + tracing::error!(error = %e, "load_table: failed to load asset"); + return internal("Failed to load table"); } }; @@ -597,13 +695,10 @@ pub async fn load_table( match check_permission(&store, &session, &Action::Read, &scope).await { Ok(true) => (), - Ok(false) => return (StatusCode::FORBIDDEN, "Forbidden").into_response(), + Ok(false) => return forbidden("Forbidden"), Err(e) => { - return ( - StatusCode::INTERNAL_SERVER_ERROR, - format!("Permission check failed: {}", e), - ) - .into_response() + tracing::error!(error = %e, "load_table: permission check failed"); + return internal("Permission check failed"); } } @@ -612,12 +707,9 @@ pub async fn load_table( if let Some(location) = current_metadata_location { let metadata_bytes = match store.read_file(&location).await { Ok(bytes) => bytes, - Err(_) => { - return ( - StatusCode::INTERNAL_SERVER_ERROR, - "Failed to read metadata file", - ) - .into_response() + Err(e) => { + tracing::error!(error = %e, "load_table: failed to read metadata file"); + return internal("Failed to read metadata file"); } }; @@ -629,15 +721,13 @@ pub async fn load_table( .await { Ok(Ok(m)) => m, - Ok(Err(_)) => { - return ( - StatusCode::INTERNAL_SERVER_ERROR, - "Failed to parse metadata", - ) - .into_response() + Ok(Err(e)) => { + tracing::error!(error = %e, "update_table: failed to parse metadata"); + return internal("Failed to parse metadata"); } - Err(_) => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Task join error").into_response() + Err(e) => { + tracing::error!(error = %e, "update_table: metadata parse task panicked"); + return internal("Failed to parse metadata"); } }; @@ -683,7 +773,7 @@ pub async fn load_table( ) .into_response() } else { - (StatusCode::NOT_FOUND, "Metadata location not found").into_response() + no_such_table(&format!("{}.{}", ns_vec.join("."), tbl_name)) } } @@ -736,15 +826,23 @@ pub async fn update_table( let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { Ok(Some(c)) => c, - Ok(None) => return (StatusCode::NOT_FOUND, "Catalog not found").into_response(), - Err(_) => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response() + Ok(None) => return no_such_namespace(&catalog_name), + Err(e) => { + tracing::error!(error = %e, "update_table: failed to load catalog"); + return internal("Failed to load catalog"); } }; + // B16a: this used to split the namespace on 0x1F while `create_table` and + // `load_table` went through `parse_table_identifier` (a *single*-element + // namespace). A table created in namespace `a\x1Fb` was registered under + // `["a\x1Fb"]` and looked up here under `["a", "b"]`, so every commit to a + // nested namespace 404'd and the CAS loop below never ran at all. let (table_name, branch_from_name) = parse_table_identifier(&table); - let branch = branch_from_name.unwrap_or("main".to_string()); - let namespace_parts: Vec = namespace.split('\x1F').map(|s| s.to_string()).collect(); + let (namespace_parts, branch_from_ns) = parse_namespace(&namespace); + let branch = branch_from_name + .or(branch_from_ns) + .unwrap_or_else(|| "main".to_string()); let asset = match store .get_asset( @@ -757,9 +855,10 @@ pub async fn update_table( .await { Ok(Some(a)) => a, - Ok(None) => return (StatusCode::NOT_FOUND, "Table not found").into_response(), - Err(_) => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response() + Ok(None) => return no_such_table(&format!("{}.{}", namespace_parts.join("."), table_name)), + Err(e) => { + tracing::error!(error = %e, "update_table: failed to load asset"); + return internal("Failed to load table"); } }; @@ -771,13 +870,10 @@ pub async fn update_table( match check_permission(&store, &session, &Action::Write, &scope).await { Ok(true) => (), - Ok(false) => return (StatusCode::FORBIDDEN, "Forbidden").into_response(), + Ok(false) => return forbidden("Forbidden"), Err(e) => { - return ( - StatusCode::INTERNAL_SERVER_ERROR, - format!("Permission check failed: {}", e), - ) - .into_response() + tracing::error!(error = %e, "update_table: permission check failed"); + return internal("Permission check failed"); } } @@ -798,9 +894,12 @@ pub async fn update_table( .await { Ok(Some(a)) => a, - Ok(None) => return (StatusCode::NOT_FOUND, "Table not found").into_response(), - Err(_) => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response() + Ok(None) => { + return no_such_table(&format!("{}.{}", namespace_parts.join("."), table_name)) + } + Err(e) => { + tracing::error!(error = %e, "update_table: failed to re-read asset"); + return internal("Failed to load table"); } }; @@ -809,16 +908,13 @@ pub async fn update_table( let metadata_bytes = if let Some(loc) = ¤t_metadata_location { match store.read_file(loc).await { Ok(bytes) => bytes, - Err(_) => { - return (StatusCode::NOT_FOUND, "Failed to read metadata file").into_response() + Err(e) => { + tracing::error!(error = %e, "update_table: failed to read metadata file"); + return internal("Failed to read metadata file"); } } } else { - return ( - StatusCode::INTERNAL_SERVER_ERROR, - "Table corrupted (no metadata)", - ) - .into_response(); + return internal("Table corrupted (no metadata location)"); }; // Parse metadata in a blocking task to avoid stalling the executor @@ -829,15 +925,13 @@ pub async fn update_table( .await { Ok(Ok(m)) => m, - Ok(Err(_)) => { - return ( - StatusCode::INTERNAL_SERVER_ERROR, - "Failed to parse metadata", - ) - .into_response() + Ok(Err(e)) => { + tracing::error!(error = %e, "load_table: failed to parse metadata"); + return internal("Failed to parse metadata"); } - Err(_) => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Task join error").into_response() + Err(e) => { + tracing::error!(error = %e, "load_table: metadata parse task panicked"); + return internal("Failed to parse metadata"); } }; @@ -854,6 +948,36 @@ pub async fn update_table( return commit_error_response(e); } + // B16b: `last-updated-ms` was only ever assigned inside `add_snapshot`, + // so a commit of only `set-properties` / `add-schema` / `set-location` / + // `add-spec` / `set-snapshot-ref` / `remove-snapshots` published a new + // metadata file carrying an *unchanged* timestamp - and any consumer + // that orders or dedupes metadata by that field treated the two versions + // as identical. Every successful set of updates bumps it. + metadata.last_updated_ms = Utc::now().timestamp_millis(); + + // B13: record the metadata file this one supersedes. `metadata-log` was + // initialised to an empty vec at table creation and never appended to, + // so metadata time-travel and previous-version cleanup + // (`write.metadata.previous-versions-max`) had nothing to work with. + if let Some(previous) = ¤t_metadata_location { + let log = metadata.metadata_log.get_or_insert_with(Vec::new); + log.push(MetadataLogEntry { + timestamp_ms: metadata.last_updated_ms, + metadata_file: previous.clone(), + }); + let max_entries = metadata + .properties + .as_ref() + .and_then(|p| p.get("write.metadata.previous-versions-max")) + .and_then(|v| v.parse::().ok()) + .unwrap_or(DEFAULT_PREVIOUS_VERSIONS_MAX); + if log.len() > max_entries { + let excess = log.len() - max_entries; + log.drain(0..excess); + } + } + let new_metadata_location = format!( "{}/metadata/00000-{}.metadata.json", metadata.location, @@ -863,23 +987,16 @@ pub async fn update_table( Ok(json) => json, Err(e) => { tracing::error!(error = %e, "could not serialise table metadata"); - return ( - StatusCode::INTERNAL_SERVER_ERROR, - "Failed to serialise table metadata", - ) - .into_response(); + return internal("Failed to serialise table metadata"); } }; - if let Err(_) = store + if store .write_file(&new_metadata_location, metadata_json.into_bytes()) .await + .is_err() { - return ( - StatusCode::INTERNAL_SERVER_ERROR, - "Failed to write new metadata", - ) - .into_response(); + return internal("Failed to write new metadata"); } match store @@ -938,6 +1055,19 @@ pub async fn update_table( // Re-read and re-check requirements on the next pass. tracing::debug!(error = %e, attempt = retries, "metadata CAS lost, retrying"); crate::metrics::inc(&crate::metrics::COMMIT_CAS_RETRIES); + // B16d: the metadata file was written *before* the CAS, so on a + // lost CAS it is unreferenced - and the old code just + // `continue`d, orphaning it. Under contention that leaked up to + // one file per retry, with up to five left behind on a final + // give-up, and an orphan is indistinguishable from live metadata + // from the outside so nothing could reap it later. + if let Err(cleanup) = store.delete_file(&new_metadata_location).await { + tracing::warn!( + error = %cleanup, + location = %new_metadata_location, + "could not remove the metadata file orphaned by a lost CAS" + ); + } retries += 1; continue; } @@ -974,6 +1104,13 @@ fn commit_error_response(error: commit::CommitError) -> axum::response::Response } /// Rename a table +/// +/// B0c: this handler bound `Extension(session)` but never called +/// `check_permission` - the only table handler that didn't. Any tenant member +/// could move any table into any namespace: an effective delete (the table +/// vanishes from where its readers look for it) and a way to smuggle a table +/// into a namespace where the caller *does* have read rights. It now needs +/// `Write` on the source table and `Create` on the destination namespace. #[utoipa::path( post, path = "/v1/{prefix}/tables/rename", @@ -984,6 +1121,7 @@ fn commit_error_response(error: commit::CommitError) -> axum::response::Response request_body = RenameTableRequest, responses( (status = 204, description = "Table renamed"), + (status = 403, description = "Forbidden"), (status = 404, description = "Source table not found"), (status = 500, description = "Internal server error") ), @@ -1022,6 +1160,85 @@ pub async fn rename_table( let dest_name = payload.destination.name; let branch = Some("main".to_string()); + let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { + Ok(Some(c)) => c, + Ok(None) => return no_such_namespace(&catalog_name), + Err(e) => { + tracing::error!(error = %e, "rename_table: failed to load catalog"); + return internal("Failed to load catalog"); + } + }; + + let source_asset = match store + .get_asset( + tenant_id, + &catalog_name, + branch.clone(), + source_ns.clone(), + source_name.clone(), + ) + .await + { + Ok(Some(a)) => a, + Ok(None) => return no_such_table(&format!("{}.{}", source_ns.join("."), source_name)), + Err(e) => { + tracing::error!(error = %e, "rename_table: failed to load source asset"); + return internal("Failed to load source table"); + } + }; + + // Write on the source: renaming is a mutation of the table's identity, and + // from every reader's point of view it is a delete at the old path. + let source_scope = PermissionScope::Asset { + catalog_id: catalog.id, + namespace: source_ns.join("."), + asset_id: source_asset.id, + }; + match check_permission(&store, &session, &Action::Write, &source_scope).await { + Ok(true) => (), + Ok(false) => return forbidden("Forbidden: no write access to the source table"), + Err(e) => { + tracing::error!(error = %e, "rename_table: source permission check failed"); + return internal("Permission check failed"); + } + } + + // Create on the destination namespace: otherwise a caller with write on one + // table could plant it anywhere they can read. + let dest_scope = PermissionScope::Namespace { + catalog_id: catalog.id, + namespace: dest_ns.join("."), + }; + match check_permission(&store, &session, &Action::Create, &dest_scope).await { + Ok(true) => (), + Ok(false) => return forbidden("Forbidden: no create access in the destination namespace"), + Err(e) => { + tracing::error!(error = %e, "rename_table: destination permission check failed"); + return internal("Permission check failed"); + } + } + + // The spec returns 409 rather than clobbering an existing destination. + match store + .get_asset( + tenant_id, + &catalog_name, + branch.clone(), + dest_ns.clone(), + dest_name.clone(), + ) + .await + { + Ok(Some(_)) => { + return table_already_exists(&format!("{}.{}", dest_ns.join("."), dest_name)) + } + Ok(None) => {} + Err(e) => { + tracing::error!(error = %e, "rename_table: destination existence check failed"); + return internal("Failed to check the destination table"); + } + } + match store .rename_asset( tenant_id, @@ -1044,7 +1261,7 @@ pub async fn rename_table( session.username.clone(), pangolin_core::audit::AuditAction::RenameTable, pangolin_core::audit::ResourceType::Table, - None, // Cannot determine asset ID easily without lookup + Some(source_asset.id), format!( "{}/{}.{} -> {}.{}", catalog_name, @@ -1059,7 +1276,10 @@ pub async fn rename_table( StatusCode::NO_CONTENT.into_response() } - Err(_) => (StatusCode::NOT_FOUND, "Table not found").into_response(), + Err(e) => { + tracing::error!(error = %e, "rename_table: rename failed"); + no_such_table(&format!("{}.{}", source_ns.join("."), source_name)) + } } } @@ -1107,15 +1327,18 @@ pub async fn delete_table( let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { Ok(Some(c)) => c, - Ok(None) => return (StatusCode::NOT_FOUND, "Catalog not found").into_response(), - Err(_) => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Internal server error").into_response() + Ok(None) => return no_such_namespace(&catalog_name), + Err(e) => { + tracing::error!(error = %e, "delete_table: failed to load catalog"); + return internal("Failed to load catalog"); } }; let (table_name, branch_from_name) = parse_table_identifier(&table); - let branch = branch_from_name.or(Some("main".to_string())); - let namespace_parts: Vec = namespace.split('\x1F').map(|s| s.to_string()).collect(); + let (namespace_parts, branch_from_ns) = parse_namespace(&namespace); + let branch = branch_from_name + .or(branch_from_ns) + .or(Some("main".to_string())); let asset = match store .get_asset( @@ -1128,9 +1351,10 @@ pub async fn delete_table( .await { Ok(Some(a)) => a, - Ok(None) => return (StatusCode::NOT_FOUND, "Table not found").into_response(), - Err(_) => { - return (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response() + Ok(None) => return no_such_table(&format!("{}.{}", namespace_parts.join("."), table_name)), + Err(e) => { + tracing::error!(error = %e, "delete_table: failed to load asset"); + return internal("Failed to load table"); } }; @@ -1142,13 +1366,10 @@ pub async fn delete_table( match check_permission(&store, &session, &Action::Delete, &scope).await { Ok(true) => (), - Ok(false) => return (StatusCode::FORBIDDEN, "Forbidden").into_response(), + Ok(false) => return forbidden("Forbidden"), Err(e) => { - return ( - StatusCode::INTERNAL_SERVER_ERROR, - format!("Permission check failed: {}", e), - ) - .into_response() + tracing::error!(error = %e, "delete_table: permission check failed"); + return internal("Permission check failed"); } } @@ -1172,7 +1393,7 @@ pub async fn delete_table( session.username.clone(), pangolin_core::audit::AuditAction::DropTable, pangolin_core::audit::ResourceType::Table, - None, + Some(asset.id), format!("{}/{}/{}", catalog_name, namespace, table), ), ) @@ -1180,7 +1401,10 @@ pub async fn delete_table( StatusCode::NO_CONTENT.into_response() } - Err(_) => (StatusCode::INTERNAL_SERVER_ERROR, "Internal Server Error").into_response(), + Err(e) => { + tracing::error!(error = %e, "delete_table: delete failed"); + internal("Failed to delete table") + } } } @@ -1203,6 +1427,7 @@ pub async fn delete_table( pub async fn table_exists( State(store): State, Extension(tenant): Extension, + Extension(session): Extension, Path((prefix, namespace, table)): Path<(String, String, String)>, ) -> impl IntoResponse { let tenant_id = tenant.0; @@ -1223,26 +1448,52 @@ pub async fn table_exists( return response; } - let namespace_parts: Vec = namespace.split('\x1F').map(|s| s.to_string()).collect(); - let (table_name, branch_name) = if let Some((t, b)) = table.split_once('@') { - (t.to_string(), Some(b.to_string())) - } else { - (table.to_string(), None) + let (namespace_parts, branch_from_ns) = parse_namespace(&namespace); + let (table_name, branch_from_table) = parse_table_identifier(&table); + let branch_name = branch_from_table.or(branch_from_ns); + + let catalog = match store.get_catalog(tenant_id, catalog_name.clone()).await { + Ok(Some(c)) => c, + Ok(None) => return StatusCode::NOT_FOUND.into_response(), + Err(e) => { + tracing::error!(error = %e, "table_exists: failed to load catalog"); + return StatusCode::INTERNAL_SERVER_ERROR.into_response(); + } }; - match store + let asset = match store .get_asset( tenant_id, &catalog_name, branch_name, - namespace_parts, + namespace_parts.clone(), table_name, ) .await { - Ok(Some(_)) => StatusCode::OK.into_response(), - Ok(None) => StatusCode::NOT_FOUND.into_response(), - Err(_) => StatusCode::INTERNAL_SERVER_ERROR.into_response(), + Ok(Some(a)) => a, + Ok(None) => return StatusCode::NOT_FOUND.into_response(), + Err(e) => { + tracing::error!(error = %e, "table_exists: failed to load asset"); + return StatusCode::INTERNAL_SERVER_ERROR.into_response(); + } + }; + + // Existence is information. Without a check this endpoint is an oracle that + // reports whether a table the caller cannot read exists - every sibling + // handler gates on Read, so this one does too. + let scope = PermissionScope::Asset { + catalog_id: catalog.id, + namespace: namespace_parts.join("."), + asset_id: asset.id, + }; + match check_permission(&store, &session, &Action::Read, &scope).await { + Ok(true) => StatusCode::OK.into_response(), + Ok(false) => StatusCode::NOT_FOUND.into_response(), + Err(e) => { + tracing::error!(error = %e, "table_exists: permission check failed"); + StatusCode::INTERNAL_SERVER_ERROR.into_response() + } } } diff --git a/pangolin/pangolin_api/src/iceberg/types.rs b/pangolin/pangolin_api/src/iceberg/types.rs index d792784..2e6999f 100644 --- a/pangolin/pangolin_api/src/iceberg/types.rs +++ b/pangolin/pangolin_api/src/iceberg/types.rs @@ -12,6 +12,9 @@ pub struct CatalogConfig { #[derive(Serialize, ToSchema)] pub struct ListNamespacesResponse { pub namespaces: Vec>, + /// Continuation token; absent on the final page (B16i). + #[serde(rename = "next-page-token", skip_serializing_if = "Option::is_none")] + pub next_page_token: Option, } #[derive(Deserialize, IntoParams)] @@ -107,6 +110,13 @@ impl TableResponse { #[derive(Serialize, Deserialize, ToSchema)] pub struct ListTablesResponse { pub identifiers: Vec, + /// Continuation token; absent on the final page (B16i). + #[serde( + rename = "next-page-token", + skip_serializing_if = "Option::is_none", + default + )] + pub next_page_token: Option, } #[derive(Serialize, Deserialize, ToSchema)] @@ -258,6 +268,108 @@ pub fn parse_table_identifier(identifier: &str) -> (String, Option) { } } +/// The unit separator the Iceberg REST spec uses to encode a multi-level +/// namespace inside a single path segment. +pub const NAMESPACE_SEPARATOR: char = '\u{1F}'; + +/// Parse a namespace path segment into its levels plus an optional branch. +/// +/// This is the single parser for namespace path segments (B16a). Handlers used +/// to disagree: `list_tables`/`create_table`/`load_table` went through +/// [`parse_table_identifier`], which yields a *single-element* namespace, while +/// `update_table`/`delete_table`/`table_exists` split on `0x1F` and yielded the +/// real multi-element path. So a table created in namespace `a\x1Fb` was +/// registered under `["a\x1Fb"]` but looked up under `["a", "b"]` on commit - +/// a guaranteed `404 Table not found`, with the CAS loop never running. +/// +/// The `@branch` suffix is stripped first so a `ns@branch` form works +/// everywhere, not just on the handlers that happened to call +/// `parse_table_identifier`. +pub fn parse_namespace(namespace: &str) -> (Vec, Option) { + let (path, branch) = match namespace.split_once('@') { + Some((path, branch)) if !branch.is_empty() => (path, Some(branch.to_string())), + _ => (namespace, None), + }; + + let levels: Vec = path + .split(NAMESPACE_SEPARATOR) + .filter(|s| !s.is_empty()) + .map(|s| s.to_string()) + .collect(); + + (levels, branch) +} + +/// Spec-shaped pagination query parameters. +/// +/// The Iceberg REST spec paginates with `pageToken`/`pageSize`; Pangolin only +/// understood `limit`/`offset`, so a spec client's paging parameters were +/// silently ignored and it had no way to detect a truncated listing (B16i). +/// Both spellings are accepted, with the spec ones taking precedence. +#[derive(Deserialize, IntoParams, Default)] +pub struct IcebergPageParams { + #[serde(rename = "pageToken")] + pub page_token: Option, + #[serde(rename = "pageSize")] + pub page_size: Option, + pub limit: Option, + pub offset: Option, +} + +/// Default page size when a client asks for pagination without naming one. +pub const DEFAULT_PAGE_SIZE: u32 = 100; + +impl IcebergPageParams { + /// Resolve to `(offset, limit)`. + /// + /// The page token is an opaque encoding of the offset, per the spec's + /// "clients must treat the token as opaque" rule; the encoding here is just + /// a prefixed decimal so it stays debuggable, and an unparseable token + /// degrades to offset 0 rather than erroring the listing. + pub fn resolve(&self) -> (u32, u32) { + let limit = self + .page_size + .or(self.limit) + .filter(|l| *l > 0) + .unwrap_or(DEFAULT_PAGE_SIZE); + + let offset = self + .page_token + .as_deref() + .and_then(decode_page_token) + .or(self.offset) + .unwrap_or(0); + + (offset, limit) + } +} + +/// Encode an offset as an opaque continuation token. +pub fn encode_page_token(offset: u32) -> String { + use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine as _}; + URL_SAFE_NO_PAD.encode(format!("o:{}", offset)) +} + +/// Decode a continuation token produced by [`encode_page_token`]. +pub fn decode_page_token(token: &str) -> Option { + use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine as _}; + let decoded = URL_SAFE_NO_PAD.decode(token).ok()?; + let decoded = String::from_utf8(decoded).ok()?; + decoded.strip_prefix("o:")?.parse().ok() +} + +/// Compute the `next-page-token` for a listing. +/// +/// Returns `None` when the page came back short, which is how a client knows it +/// has reached the end. +pub fn next_page_token(returned: usize, offset: u32, limit: u32) -> Option { + if returned as u32 == limit { + Some(encode_page_token(offset + limit)) + } else { + None + } +} + #[derive(Deserialize, Serialize, ToSchema)] pub struct RenameTableRequest { pub source: TableIdentifier, diff --git a/pangolin/pangolin_api/src/lib.rs b/pangolin/pangolin_api/src/lib.rs index e44e056..60c8e3b 100644 --- a/pangolin/pangolin_api/src/lib.rs +++ b/pangolin/pangolin_api/src/lib.rs @@ -151,8 +151,14 @@ pub fn app_with_options( get(iceberg::namespaces::list_namespaces).post(iceberg::namespaces::create_namespace), ) .route( + // `loadNamespaceMetadata` and `namespaceExists` were on the README's + // "not implemented" list: a client could create a namespace and set + // properties but never read them back, and had no cheap existence + // probe. "/v1/:prefix/namespaces/:namespace", - delete(iceberg::namespaces::delete_namespace), + get(iceberg::namespaces::load_namespace_metadata) + .head(iceberg::namespaces::namespace_exists) + .delete(iceberg::namespaces::delete_namespace), ) .route( "/v1/:prefix/namespaces/:namespace/properties", @@ -192,7 +198,9 @@ pub fn app_with_options( ) .route( "/v1/:prefix/v1/namespaces/:namespace", - delete(iceberg::namespaces::delete_namespace), + get(iceberg::namespaces::load_namespace_metadata) + .head(iceberg::namespaces::namespace_exists) + .delete(iceberg::namespaces::delete_namespace), ) .route( "/v1/:prefix/v1/namespaces/:namespace/properties", @@ -558,15 +566,33 @@ pub fn app_with_options( // Resource safety. There were previously no limits of any kind (A-20): // a single large POST could be buffered without bound and a slow // backend request had no deadline. + // + // Layer order matters, and `.layer()` applies *outermost last*, so this + // list reads inside-out: body limit is innermost, then the concurrency + // limiter, then the timeout, then load shedding. + // + // B16l: the timeout used to sit *inside* the concurrency limiter, so a + // request queued for one of the permits had no deadline at all - the + // 30s clock only started once it was admitted. Under sustained overload + // the queue grew without bound and clients saw latencies far past + // `PANGOLIN_REQUEST_TIMEOUT_SECS`. With the timeout outside, the + // deadline covers queueing time, which is what a client's timeout budget + // actually cares about. .layer(DefaultBodyLimit::max(options.body_limit_bytes)) - .layer(tower_http::timeout::TimeoutLayer::new( - options.request_timeout, - )) // GlobalConcurrencyLimitLayer rather than ConcurrencyLimitLayer: the // latter's service is not `Clone`, which axum requires. .layer(tower::limit::GlobalConcurrencyLimitLayer::new( options.concurrency_limit, )) + .layer(tower_http::timeout::TimeoutLayer::new( + options.request_timeout, + )) + // B0m: token issuance could be driven to panic by a request-controlled + // `expires_in_hours`, and with no catch-panic layer the panic tore down + // the whole connection task rather than failing one request. The + // arithmetic is now total, but a panic anywhere else should still be a + // 500 for one caller rather than a dropped connection for several. + .layer(tower_http::catch_panic::CatchPanicLayer::new()) // Request IDs, access logging and metrics. `tower-http` was already // built with the `trace` feature but `TraceLayer` was never applied. .layer(axum::middleware::from_fn(observability::track_request)) diff --git a/pangolin/pangolin_api/src/main.rs b/pangolin/pangolin_api/src/main.rs index 18d2d1a..960e6a4 100644 --- a/pangolin/pangolin_api/src/main.rs +++ b/pangolin/pangolin_api/src/main.rs @@ -114,11 +114,25 @@ async fn main() { // the compare-and-swap that publishes it leaks an orphaned metadata file. let serve = axum::serve(listener, app).with_graceful_shutdown(shutdown_signal(shutdown_grace)); - if let Err(e) = serve.await { - tracing::error!("server error: {e}"); - std::process::exit(1); + // B16n: `shutdown_grace` used to be logged and nothing else. There was no + // bound on the drain at all, so `with_graceful_shutdown` waited for + // in-flight connections *indefinitely* - one hung upstream call blocked + // SIGTERM past the k8s termination grace period and into a SIGKILL, which + // is exactly the mid-commit kill this whole path exists to avoid. + // `PANGOLIN_SHUTDOWN_GRACE_SECS` now actually bounds the drain. + match tokio::time::timeout(shutdown_grace, serve).await { + Ok(Ok(())) => tracing::info!("shutdown complete"), + Ok(Err(e)) => { + tracing::error!("server error: {e}"); + std::process::exit(1); + } + Err(_) => { + tracing::warn!( + grace_secs = shutdown_grace.as_secs(), + "drain did not finish within the shutdown grace period; exiting anyway" + ); + } } - tracing::info!("shutdown complete"); } /// Probe this instance's readiness endpoint. Returns a process exit code. @@ -367,4 +381,12 @@ async fn shutdown_signal(grace: std::time::Duration) { // while in-flight requests finish. health::mark_draining(); tracing::info!(grace_secs = grace.as_secs(), "draining in-flight requests"); + + // Give the load balancer a moment to observe the failing readiness probe + // before the listener stops accepting. Without this pause the LB can still + // be routing new connections at the instant we stop accepting them, which + // shows up to clients as connection resets during every rolling update. + // Capped so it can never consume the whole grace budget. + let deregistration_pause = std::cmp::min(grace / 4, std::time::Duration::from_secs(5)); + tokio::time::sleep(deregistration_pause).await; } diff --git a/pangolin/pangolin_api/src/oauth_handlers.rs b/pangolin/pangolin_api/src/oauth_handlers.rs index a3bf06e..2be5cbd 100644 --- a/pangolin/pangolin_api/src/oauth_handlers.rs +++ b/pangolin/pangolin_api/src/oauth_handlers.rs @@ -25,6 +25,50 @@ pub struct OAuthUserInfo { pub sub: String, pub email: String, pub name: Option, + /// Whether the provider asserts the address has been verified. + /// + /// Absent on providers that do not report it - notably GitHub's + /// `/user` endpoint - which is why an absent value is treated as + /// *unverified* (B0l). + #[serde(default)] + pub email_verified: Option, +} + +impl OAuthUserInfo { + fn email_is_verified(&self) -> bool { + self.email_verified.unwrap_or(false) + } +} + +/// Domains whose *verified* addresses may link to a pre-existing local account. +/// +/// Empty by default: with no allowlist configured, email never links an account, +/// and only a `(provider, subject)` match does. +fn email_link_domain_allowlist() -> Vec { + std::env::var("PANGOLIN_OAUTH_EMAIL_LINK_DOMAINS") + .ok() + .map(|v| { + v.split(',') + .map(|s| s.trim().to_ascii_lowercase()) + .filter(|s| !s.is_empty()) + .collect() + }) + .unwrap_or_default() +} + +/// May `email` be used to adopt an existing local account? +fn email_may_link(user_info: &OAuthUserInfo, allowlist: &[String]) -> bool { + if !user_info.email_is_verified() { + return false; + } + let Some(domain) = user_info + .email + .rsplit_once('@') + .map(|(_, d)| d.to_ascii_lowercase()) + else { + return false; + }; + allowlist.iter().any(|allowed| *allowed == domain) } #[derive(Deserialize, ToSchema)] @@ -180,23 +224,32 @@ pub async fn oauth_callback( }; // 4. Find or Create User - // We assume email is unique and can be used to link accounts or create new ones - // For MVP, we'll create a new user if not found by email, or maybe by oauth_subject - // Ideally look up by (provider, subject) - - // Since CatalogStore doesn't expose `get_user_by_oauth` yet, we'll use `get_user_by_username` as a fallback or iterate - // But `CatalogStore` trait needs a method for this efficiently. - // For now, let's list users and filter (inefficient but works for MemoryStore) - // Or better, let's just stick to email for now if unique. - - // Let's implement a rudimentary lookup + // + // B0l: the match used to include `|| u.email == user_info.email`, with no + // `email_verified` check and no provider binding. Anyone who could set a + // matching address on *any* configured provider - GitHub happily reports + // unverified addresses - logged in as that Pangolin user, including the + // seeded `TenantAdmin`. Identity is `(provider, subject)`; an address is + // only allowed to adopt a pre-existing account when the provider says it is + // verified *and* its domain is one the operator listed. let all_users = store.list_users(None, None).await.unwrap_or_default(); + let allowlist = email_link_domain_allowlist(); + let may_link_by_email = email_may_link(&user_info, &allowlist); + let existing_user = all_users.into_iter().find(|u| { - (u.oauth_provider == Some(provider_enum.clone()) - && u.oauth_subject == Some(user_info.sub.clone())) - || u.email == user_info.email + let subject_match = u.oauth_provider == Some(provider_enum.clone()) + && u.oauth_subject == Some(user_info.sub.clone()); + let email_match = may_link_by_email && u.email == user_info.email; + subject_match || email_match }); + if !may_link_by_email && !allowlist.is_empty() && !user_info.email_is_verified() { + tracing::warn!( + provider = %provider, + "OAuth provider reported an unverified email; not linking by address" + ); + } + let user = match existing_user { Some(u) => { // Update last login or details if needed diff --git a/pangolin/pangolin_api/src/optimization_handlers.rs b/pangolin/pangolin_api/src/optimization_handlers.rs index 63358fd..1ee16df 100644 --- a/pangolin/pangolin_api/src/optimization_handlers.rs +++ b/pangolin/pangolin_api/src/optimization_handlers.rs @@ -106,8 +106,13 @@ pub async fn search_assets_by_name( catalogs.iter().map(|c| (c.name.clone(), c.id)).collect(); // Apply permission-based filtering - let filtered_assets = - crate::authz_utils::filter_assets(assets, &permissions, session.role.clone(), &catalog_map); + let filtered_assets = crate::authz_utils::filter_assets( + tenant_id, + assets, + &permissions, + session.role.clone(), + &catalog_map, + ); // Filter by catalog if specified let mut all_results = Vec::new(); @@ -128,6 +133,7 @@ pub async fn search_assets_by_name( let namespace_str = namespace.join("."); let required_actions = vec![pangolin_core::permission::Action::Read]; crate::authz_utils::has_asset_access( + tenant_id, catalog_id, &namespace_str, asset.id, @@ -401,8 +407,12 @@ pub async fn unified_search( .search_catalogs(tenant_id, &query.q) .await .map_err(ApiError::from)?; - let filtered_catalogs = - crate::authz_utils::filter_catalogs(catalogs, &permissions, session.role.clone()); + let filtered_catalogs = crate::authz_utils::filter_catalogs( + tenant_id, + catalogs, + &permissions, + session.role.clone(), + ); for c in filtered_catalogs { results.push(UnifiedSearchResult { id: Some(c.id.to_string()), @@ -430,6 +440,7 @@ pub async fn unified_search( .collect(); let filtered_namespaces = crate::authz_utils::filter_namespaces( + tenant_id, namespaces, &permissions, session.role.clone(), @@ -452,6 +463,7 @@ pub async fn unified_search( .await .map_err(ApiError::from)?; let filtered_assets = crate::authz_utils::filter_assets( + tenant_id, assets, &permissions, session.role.clone(), @@ -481,6 +493,7 @@ pub async fn unified_search( session.role, pangolin_core::user::UserRole::Root | pangolin_core::user::UserRole::TenantAdmin ) || crate::authz_utils::has_catalog_access( + tenant_id, catalog_id, &permissions, &[pangolin_core::permission::Action::Read], diff --git a/pangolin/pangolin_api/src/pangolin_handlers.rs b/pangolin/pangolin_api/src/pangolin_handlers.rs index 1cd126c..e994ef5 100644 --- a/pangolin/pangolin_api/src/pangolin_handlers.rs +++ b/pangolin/pangolin_api/src/pangolin_handlers.rs @@ -769,8 +769,12 @@ pub async fn list_catalogs( Vec::new() // Root/TenantAdmin bypass filtering }; - let filtered_catalogs = - crate::authz_utils::filter_catalogs(catalogs, &permissions, session.role.clone()); + let filtered_catalogs = crate::authz_utils::filter_catalogs( + tenant_id, + catalogs, + &permissions, + session.role.clone(), + ); tracing::info!( "list_catalogs returning {} catalogs for user {}", diff --git a/pangolin/pangolin_api/src/public_paths.rs b/pangolin/pangolin_api/src/public_paths.rs index ef128e3..c987141 100644 --- a/pangolin/pangolin_api/src/public_paths.rs +++ b/pangolin/pangolin_api/src/public_paths.rs @@ -53,6 +53,21 @@ pub fn is_public_path(path: &str) -> bool { ["oauth", "authorize", _provider] => true, ["oauth", "callback", _provider] => true, + // Redeeming the one-time code the OAuth callback hands back. + // + // B0k: this was missing, so the middleware demanded a bearer token on + // the very endpoint whose job is to obtain the first one. The 0.6.0 + // callback -> one-time code -> POST exchange flow was therefore + // unreachable in production: the browser landed with a `?code=...` it + // could never redeem. The code itself is single-use and short-lived, + // which is what makes this endpoint safe to expose unauthenticated. + ["api", "v1", "oauth", "exchange"] => true, + + // Which OAuth providers are configured. The login page needs this + // before anyone is authenticated (see B33); it reveals only provider + // names, never secrets. + ["api", "v1", "oauth", "providers"] => true, + _ => false, } } @@ -87,6 +102,18 @@ mod tests { } } + /// Regression test for B0k: without this the OAuth login flow cannot + /// complete, because the code-exchange endpoint demanded the token it + /// exists to issue. + #[test] + fn oauth_exchange_and_providers_are_public() { + assert!(is_public_path("/api/v1/oauth/exchange")); + assert!(is_public_path("/api/v1/oauth/providers")); + // ...but nothing deeper under the same prefix. + assert!(!is_public_path("/api/v1/oauth/exchange/steal")); + assert!(!is_public_path("/api/v1/oauth/tokens")); + } + /// Regression test for A-11: a resource named `config` must not bypass auth. #[test] fn resources_named_config_are_not_public() { diff --git a/pangolin/pangolin_api/src/signing_handlers.rs b/pangolin/pangolin_api/src/signing_handlers.rs index f0c2f7b..70010f8 100644 --- a/pangolin/pangolin_api/src/signing_handlers.rs +++ b/pangolin/pangolin_api/src/signing_handlers.rs @@ -1,5 +1,6 @@ use crate::auth::TenantId; -use crate::iceberg::AppState; +use crate::authz::check_permission; +use crate::iceberg::{parse_namespace, parse_table_identifier, AppState}; use axum::Extension; use axum::{ extract::{Path, Query, State}, @@ -7,6 +8,8 @@ use axum::{ response::IntoResponse, Json, }; +use pangolin_core::permission::{Action, PermissionScope}; +use pangolin_core::user::UserSession; use serde::{Deserialize, Serialize}; use std::collections::HashMap; use utoipa::{IntoParams, ToSchema}; @@ -180,6 +183,17 @@ pub async fn get_gcp_token(_service_account_key_json: &str) -> Result Result Result, Extension(tenant): Extension, + Extension(session): Extension, Path((catalog_name, namespace, table)): Path<(String, String, String)>, ) -> impl IntoResponse { let tenant_id = tenant.0; @@ -218,7 +233,55 @@ pub async fn get_table_credentials( Err(e) => return (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()).into_response(), }; - // 2. Check if catalog has a warehouse + // 2. Resolve the asset the caller is asking for. Vending credentials for a + // table that does not exist is not a meaningful operation, and resolving it + // is what makes an asset-scoped permission check possible at all. + let (ns_levels, branch) = parse_namespace(&namespace); + let (table_name, branch_from_table) = parse_table_identifier(&table); + let branch = branch_from_table.or(branch); + + let asset = match store + .get_asset( + tenant_id, + &catalog_name, + branch, + ns_levels.clone(), + table_name.clone(), + ) + .await + { + Ok(Some(a)) => a, + Ok(None) => return (StatusCode::NOT_FOUND, "Table not found").into_response(), + Err(e) => return (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()).into_response(), + }; + + let scope = PermissionScope::Asset { + catalog_id: catalog.id, + namespace: ns_levels.join("."), + asset_id: asset.id, + }; + + // 3. Read is the floor: without it, no credentials at all. + match check_permission(&store, &session, &Action::Read, &scope).await { + Ok(true) => (), + Ok(false) => return (StatusCode::FORBIDDEN, "Forbidden").into_response(), + Err(e) => { + return ( + StatusCode::INTERNAL_SERVER_ERROR, + format!("Permission check failed: {}", e), + ) + .into_response() + } + } + + // Write is vended only when actually held, so a read-only principal gets + // read-only cloud credentials. + let can_write = matches!( + check_permission(&store, &session, &Action::Write, &scope).await, + Ok(true) + ); + + // 4. Check if catalog has a warehouse let warehouse_name = match catalog.warehouse_name { Some(name) => name, None => { @@ -231,16 +294,23 @@ pub async fn get_table_credentials( } }; - // 3. Get warehouse configuration + // 5. Get warehouse configuration let warehouse = match store.get_warehouse(tenant_id, warehouse_name).await { Ok(Some(wh)) => wh, Ok(None) => return (StatusCode::NOT_FOUND, "Warehouse not found").into_response(), Err(e) => return (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()).into_response(), }; - // 4. Vend credentials using the credential signer infrastructure - let resource_path = format!("{}/{}", namespace, table); - let permissions = vec!["read".to_string(), "write".to_string()]; + // 6. Vend credentials using the credential signer infrastructure + let resource_path = if ns_levels.is_empty() { + table_name.clone() + } else { + format!("{}/{}", ns_levels.join("/"), table_name) + }; + let mut permissions = vec!["read".to_string()]; + if can_write { + permissions.push("write".to_string()); + } match crate::credential_vending::vend_credentials_for_warehouse( &warehouse, diff --git a/pangolin/pangolin_api/src/token_handlers.rs b/pangolin/pangolin_api/src/token_handlers.rs index 83db728..407a5df 100644 --- a/pangolin/pangolin_api/src/token_handlers.rs +++ b/pangolin/pangolin_api/src/token_handlers.rs @@ -15,6 +15,7 @@ use utoipa::ToSchema; use uuid::Uuid; #[derive(Deserialize, ToSchema)] +#[serde(deny_unknown_fields)] pub struct GenerateTokenRequest { pub tenant_id: String, pub username: Option, @@ -22,6 +23,38 @@ pub struct GenerateTokenRequest { pub expires_in_hours: Option, } +/// Upper bound on a caller-requested token lifetime. +/// +/// `expires_in_hours` is attacker-controlled and used to be fed straight into +/// `chrono::Duration::hours`, which *panics* on a large enough value - before +/// the `checked_add_signed().unwrap()` below it could even run (B0m). There is +/// no `CatchPanicLayer` under the router, so that panic aborted the connection +/// task. Clamping first makes the arithmetic total. +const MAX_TOKEN_LIFETIME_HOURS: u64 = 24 * 365; + +/// Rank roles so a caller cannot mint a token more privileged than itself. +fn role_rank(role: &UserRole) -> u8 { + match role { + UserRole::Root => 3, + UserRole::TenantAdmin => 2, + UserRole::TenantUser => 1, + } +} + +/// Parse a role name from a token request. +/// +/// Accepts both the Debug-ish spellings the old code matched and the kebab-case +/// serde names, but unknown values are now an error rather than a silent +/// downgrade to `TenantUser`. +fn parse_role(name: &str) -> Option { + match name { + "Root" | "root" => Some(UserRole::Root), + "Admin" | "admin" | "TenantAdmin" | "tenant-admin" => Some(UserRole::TenantAdmin), + "TenantUser" | "tenant-user" | "user" => Some(UserRole::TenantUser), + _ => None, + } +} + #[derive(Serialize, ToSchema)] pub struct GenerateTokenResponse { pub token: String, @@ -29,8 +62,21 @@ pub struct GenerateTokenResponse { pub tenant_id: String, } -/// Generate a JWT token for a tenant -/// This endpoint allows generating tokens for testing and development +/// Generate a JWT token for a tenant. +/// +/// Authorization (B0a): this handler used to take no session at all, so any +/// authenticated principal - including the lowest-privilege `TenantUser` or any +/// service-user API key - could POST `{"tenant_id": "", "roles":["Root"]}` +/// and receive a signed `Root` token for an arbitrary tenant. That is a total +/// privilege escalation, since `check_permission` short-circuits for `Root`. +/// +/// The rules now are: +/// * `Root` may mint anything. +/// * `TenantAdmin` may mint only for its own tenant, and only a role at or +/// below its own. +/// * everyone else is refused. +/// A role supplied in the body is never trusted for a non-`Root` caller beyond +/// those bounds. #[utoipa::path( post, path = "/api/v1/tokens", @@ -39,60 +85,107 @@ pub struct GenerateTokenResponse { responses( (status = 200, description = "Token generated", body = GenerateTokenResponse), (status = 400, description = "Bad request"), + (status = 403, description = "Forbidden"), (status = 500, description = "Internal server error") - ) + ), + security(("bearer_auth" = [])) )] pub async fn generate_token( State(store): State, + Extension(session): Extension, Json(payload): Json, ) -> impl IntoResponse { // Validate tenant_id is a valid UUID - let _tenant_uuid = match Uuid::parse_str(&payload.tenant_id) { + let tenant_uuid = match Uuid::parse_str(&payload.tenant_id) { Ok(uuid) => uuid, Err(_) => return (StatusCode::BAD_REQUEST, "Invalid tenant_id format").into_response(), }; + let caller_is_root = session.role == UserRole::Root; + if !caller_is_root { + if session.role != UserRole::TenantAdmin { + return ( + StatusCode::FORBIDDEN, + "Root or tenant-admin access required to mint tokens", + ) + .into_response(); + } + if session.tenant_id != Some(tenant_uuid) { + return ( + StatusCode::FORBIDDEN, + "Cannot mint a token for another tenant", + ) + .into_response(); + } + } + let secret = crate::config::jwt_secret(); - let expires_in = payload.expires_in_hours.unwrap_or(24); + let expires_in = payload + .expires_in_hours + .unwrap_or(24) + .min(MAX_TOKEN_LIFETIME_HOURS); let now = chrono::Utc::now(); - let exp = now + let Some(exp) = now .checked_add_signed(chrono::Duration::hours(expires_in as i64)) - .unwrap() - .timestamp(); + .map(|t| t.timestamp()) + else { + return (StatusCode::BAD_REQUEST, "expires_in_hours out of range").into_response(); + }; let username = payload.username.unwrap_or_else(|| "api-user".to_string()); - // Map role strings to UserRole - // Default to lookup user role or TenantUser if not specified - let role = if let Some(roles) = &payload.roles { - if let Some(first_role) = roles.first() { - match first_role.as_str() { - "Root" | "root" => UserRole::Root, - "Admin" | "admin" | "TenantAdmin" | "tenant-admin" => UserRole::TenantAdmin, - _ => UserRole::TenantUser, - } - } else { - UserRole::TenantUser + // Map role strings to UserRole. An unknown name is now a 400 rather than a + // silent downgrade, so a typo cannot quietly hand out the wrong role. + let requested_role = if let Some(roles) = &payload.roles { + match roles.first() { + Some(first_role) => match parse_role(first_role) { + Some(r) => Some(r), + None => { + return ( + StatusCode::BAD_REQUEST, + format!("Unknown role: {}", first_role), + ) + .into_response() + } + }, + None => None, } } else { - // Try to lookup user - if let Ok(Some(user)) = store.get_user_by_username(&username).await { - tracing::info!( - "generate_token: Found user '{}' with role {:?} ({})", - username, - user.role, - user.id - ); - user.role - } else { - tracing::warn!( - "generate_token: User '{}' not found, defaulting to TenantUser", - username - ); - UserRole::TenantUser + None + }; + + let role = match requested_role { + Some(r) => r, + None => { + // Try to look up the user's own role. + if let Ok(Some(user)) = store.get_user_by_username(&username).await { + tracing::info!( + "generate_token: Found user '{}' with role {:?} ({})", + username, + user.role, + user.id + ); + user.role + } else { + tracing::warn!( + "generate_token: User '{}' not found, defaulting to TenantUser", + username + ); + UserRole::TenantUser + } } }; + // A non-root caller can never mint above its own rank, whatever the body or + // the looked-up user says. + if !caller_is_root && role_rank(&role) > role_rank(&session.role) { + return ( + StatusCode::FORBIDDEN, + "Cannot mint a token more privileged than the caller", + ) + .into_response(); + } + // sub MUST be a UUID for to_session() to work // If user exists, use their ID. Else generate one. let user_id = if let Ok(Some(user)) = store.get_user_by_username(&username).await { @@ -121,10 +214,7 @@ pub async fn generate_token( // Store token info for listing let token_info = TokenInfo { id: token_id, - tenant_id: match Uuid::parse_str(&payload.tenant_id) { - Ok(u) => u, - Err(_) => Uuid::default(), // Should be validated above - }, + tenant_id: tenant_uuid, user_id, username: username.clone(), expires_at: chrono::DateTime::from_timestamp(exp, 0).unwrap_or_default(), @@ -141,7 +231,7 @@ pub async fn generate_token( let response = GenerateTokenResponse { token, expires_at: chrono::DateTime::from_timestamp(exp, 0) - .unwrap() + .unwrap_or_default() .to_rfc3339(), tenant_id: payload.tenant_id, }; @@ -190,11 +280,22 @@ pub async fn revoke_current_token( Extension(session): Extension, Json(payload): Json, ) -> impl IntoResponse { - // Generate an expiration time (tokens typically expire in 24 hours) - let expires_at = Utc::now() + Duration::hours(24); - - // Use the user_id as the token_id for revocation - let token_id = session.user_id; + // Revoke until the token would have expired anyway; the blacklist entry + // only has to outlive the token. + let expires_at = session.expires_at.max(Utc::now()); + + // B0j: revoke the token's own `jti`. This used to revoke `session.user_id`, + // which no token ever carries as its `jti`, so the middleware's revocation + // check (keyed by `jti`) never matched: logout returned 200 and the token + // stayed valid for its full lifetime. + let Some(token_id) = session.token_id else { + // API-key and root-basic-auth sessions have no revocable JWT. + return ( + StatusCode::BAD_REQUEST, + "This session is not backed by a revocable token", + ) + .into_response(); + }; match store .revoke_token(token_id, expires_at, payload.reason) @@ -474,10 +575,16 @@ pub async fn rotate_token( let secret = crate::config::jwt_secret(); let now = chrono::Utc::now(); let expires_in = 24; // Default rotation to 24h - let exp = now + let Some(exp) = now .checked_add_signed(chrono::Duration::hours(expires_in)) - .unwrap() - .timestamp(); + .map(|t| t.timestamp()) + else { + return ( + StatusCode::INTERNAL_SERVER_ERROR, + "Failed to compute token expiry", + ) + .into_response(); + }; let token_id = Uuid::new_v4(); let tenant_id_str = session.tenant_id.map(|t| t.to_string()).unwrap_or_default(); @@ -514,21 +621,31 @@ pub async fn rotate_token( tracing::warn!("Failed to store rotated token info: {}", e); } - // 3. Revoke old token (if we knew its ID - session doesn't carry jti currently in UserSession struct?) - // Wait, UserSession struct usually just has user info. - // Current `auth_middleware` decodes claims but might not pass `jti` to `UserSession`. - // Let's check `UserSession`. If it doesn't have token ID, we can't revoke the *specific* old token easily. - // We can revoke ALL other tokens for this user? No, that's too aggressive. - // If we can't revoke the old one, "Rotation" is just "Get New Token". - // Ideally UserSession should have `token_id`. - // I'll skip revocation of old token for now if I lack the ID, but assume the client will discard it. - // Implementation: Just return new token. - // Update: We can update UserSession to include token_id (jti) later. + // 3. Revoke the old token. `UserSession` now carries the presenting + // token's `jti` (B0j), so rotation is a real rotation rather than + // "issue a second valid token and hope the client forgets the first". + if let Some(old_token_id) = session.token_id { + if let Err(e) = store + .revoke_token( + old_token_id, + session.expires_at.max(Utc::now()), + Some("Rotated".to_string()), + ) + .await + { + tracing::error!(error = %e, "failed to revoke the rotated-out token"); + return ( + StatusCode::INTERNAL_SERVER_ERROR, + "Issued a new token but could not revoke the old one", + ) + .into_response(); + } + } let response = GenerateTokenResponse { token, expires_at: chrono::DateTime::from_timestamp(exp, 0) - .unwrap() + .unwrap_or_default() .to_rfc3339(), tenant_id: tenant_id_str, }; diff --git a/pangolin/pangolin_api/tests/business_metadata_test.rs b/pangolin/pangolin_api/tests/business_metadata_test.rs index c65fed4..8e785fd 100644 --- a/pangolin/pangolin_api/tests/business_metadata_test.rs +++ b/pangolin/pangolin_api/tests/business_metadata_test.rs @@ -124,6 +124,7 @@ async fn test_business_metadata_flow() { role: UserRole::Root, issued_at: chrono::Utc::now(), expires_at: chrono::Utc::now() + chrono::Duration::hours(1), + token_id: None, }; let token = pangolin_api::auth_middleware::generate_token(session, secret).unwrap(); diff --git a/pangolin/pangolin_api/tests/credential_vending_tests.rs b/pangolin/pangolin_api/tests/credential_vending_tests.rs index fc8240c..7a80bad 100644 --- a/pangolin/pangolin_api/tests/credential_vending_tests.rs +++ b/pangolin/pangolin_api/tests/credential_vending_tests.rs @@ -233,6 +233,7 @@ fn create_test_metadata() -> TableMetadata { last_updated_ms: 1234567890, last_column_id: 1, schemas: vec![Schema { + type_: "struct".to_string(), schema_id: 0, fields: vec![], identifier_field_ids: None, @@ -242,6 +243,7 @@ fn create_test_metadata() -> TableMetadata { spec_id: 0, fields: vec![], }], + last_partition_id: 999, properties: Some(HashMap::new()), current_snapshot_id: None, snapshots: None, diff --git a/pangolin/pangolin_api/tests/iceberg_handlers_test.rs b/pangolin/pangolin_api/tests/iceberg_handlers_test.rs index ed59401..09623b2 100644 --- a/pangolin/pangolin_api/tests/iceberg_handlers_test.rs +++ b/pangolin/pangolin_api/tests/iceberg_handlers_test.rs @@ -28,6 +28,7 @@ fn test_add_snapshot_deserializes_full_object() { current_schema_id: 0, current_partition_spec_id: 0, partition_specs: vec![], + last_partition_id: 999, default_sort_order_id: 0, sort_orders: vec![], properties: None, @@ -77,6 +78,7 @@ fn test_add_snapshot_handles_multiple_snapshots() { current_schema_id: 0, current_partition_spec_id: 0, partition_specs: vec![], + last_partition_id: 999, default_sort_order_id: 0, sort_orders: vec![], properties: None, @@ -140,6 +142,7 @@ fn test_table_response_includes_config() { current_schema_id: 0, current_partition_spec_id: 0, partition_specs: vec![], + last_partition_id: 999, default_sort_order_id: 0, sort_orders: vec![], properties: None, diff --git a/pangolin/pangolin_api/tests/signing_handlers_test.rs b/pangolin/pangolin_api/tests/signing_handlers_test.rs index 246507a..a838f0d 100644 --- a/pangolin/pangolin_api/tests/signing_handlers_test.rs +++ b/pangolin/pangolin_api/tests/signing_handlers_test.rs @@ -338,6 +338,7 @@ fn test_table_response_includes_credentials() { schemas: vec![], current_schema_id: 0, partition_specs: vec![], + last_partition_id: 999, current_partition_spec_id: 0, properties: None, current_snapshot_id: None, @@ -394,6 +395,7 @@ fn test_table_response_without_credentials() { schemas: vec![], current_schema_id: 0, partition_specs: vec![], + last_partition_id: 999, current_partition_spec_id: 0, properties: None, current_snapshot_id: None, diff --git a/pangolin/pangolin_core/src/iceberg_metadata.rs b/pangolin/pangolin_core/src/iceberg_metadata.rs index f925e58..51003fb 100644 --- a/pangolin/pangolin_core/src/iceberg_metadata.rs +++ b/pangolin/pangolin_core/src/iceberg_metadata.rs @@ -3,6 +3,10 @@ use std::collections::HashMap; use utoipa::ToSchema; use uuid::Uuid; +/// Partition field ids are assigned from 1000 upward by the Iceberg spec, so an +/// unpartitioned table's highest *assigned* partition id is 999. +pub const PARTITION_FIELD_ID_START: i32 = 1000; + #[derive(Debug, Serialize, Deserialize, Clone, ToSchema)] #[serde(rename_all = "kebab-case")] pub struct TableMetadata { @@ -14,8 +18,25 @@ pub struct TableMetadata { pub last_column_id: i32, pub current_schema_id: i32, pub schemas: Vec, + /// The default partition spec id. + /// + /// The spec field is `default-spec-id`. Under the struct's kebab-case rule + /// this serialized as `current-partition-spec-id` (B11), which no + /// spec-conformant reader looks for - metadata Pangolin wrote could not be + /// read as v2 metadata by an external engine reading the file directly, and + /// a conformant engine's metadata could not round-trip in. The alias keeps + /// already-written Pangolin files parseable. + #[serde(rename = "default-spec-id", alias = "current-partition-spec-id")] pub current_partition_spec_id: i32, pub partition_specs: Vec, + /// Highest assigned partition field id. + /// + /// Required by the v2 spec and missing entirely before (B12); Java-based + /// readers reject metadata without it. Defaulted on read so files Pangolin + /// wrote earlier still parse, and recomputed from `partition_specs` by + /// [`TableMetadata::recompute_last_partition_id`]. + #[serde(default = "default_last_partition_id")] + pub last_partition_id: i32, pub default_sort_order_id: i32, pub sort_orders: Vec, pub properties: Option>, @@ -32,6 +53,28 @@ pub struct TableMetadata { pub refs: Option>, } +fn default_last_partition_id() -> i32 { + PARTITION_FIELD_ID_START - 1 +} + +impl TableMetadata { + /// Recompute `last_partition_id` from the partition specs. + /// + /// Called after any change to `partition_specs` so the field stays true; + /// the spec defines it as the highest partition field id ever assigned, so + /// it only ever moves up. + pub fn recompute_last_partition_id(&mut self) { + let highest = self + .partition_specs + .iter() + .flat_map(|spec| spec.fields.iter()) + .map(|f| f.field_id) + .max() + .unwrap_or(PARTITION_FIELD_ID_START - 1); + self.last_partition_id = self.last_partition_id.max(highest); + } +} + /// A named branch or tag pointing at a snapshot. #[derive(Debug, Serialize, Deserialize, Clone, ToSchema, PartialEq, Eq)] #[serde(rename_all = "kebab-case")] @@ -51,11 +94,79 @@ pub struct SnapshotReference { #[derive(Debug, Serialize, Deserialize, Clone, ToSchema)] #[serde(rename_all = "kebab-case")] pub struct Schema { + /// Always `"struct"`. + /// + /// Spec schemas *are* struct types and conformant writers emit this; it was + /// missing entirely (B14), so strict readers rejected the schema object. + #[serde(rename = "type", default = "struct_type_name")] + pub type_: String, + /// Defaulted because a `createTable` request body may legitimately omit it - + /// the server assigns the id. Without the default, deserializing an incoming + /// schema (which `create_table` now does instead of hand-parsing it, B16f) + /// would reject a spec-legal request. + #[serde(default)] pub schema_id: i32, + /// Omitted when absent rather than written as an explicit `null` (B14) - + /// some strict parsers reject `"identifier-field-ids": null`. + #[serde(default, skip_serializing_if = "Option::is_none")] pub identifier_field_ids: Option>, pub fields: Vec, } +fn struct_type_name() -> String { + Schema::STRUCT.to_string() +} + +impl Schema { + /// The only legal value of a schema's `type` field. + pub const STRUCT: &'static str = "struct"; + + /// Highest field id in this schema, including nested fields. + /// + /// `last-column-id` must cover nested ids too; computing it from the + /// top-level fields alone (as `create_table` used to) understates it for any + /// schema containing a struct, list or map. + pub fn max_field_id(&self) -> i32 { + fn walk(t: &Type, acc: &mut i32) { + match t { + Type::Primitive(_) => {} + Type::Struct { fields, .. } => { + for f in fields { + *acc = (*acc).max(f.id); + walk(&f.field_type, acc); + } + } + Type::List { + element_id, + element, + .. + } => { + *acc = (*acc).max(*element_id); + walk(element, acc); + } + Type::Map { + key_id, + key, + value_id, + value, + .. + } => { + *acc = (*acc).max(*key_id).max(*value_id); + walk(key, acc); + walk(value, acc); + } + } + } + + let mut max = 0; + for field in &self.fields { + max = max.max(field.id); + walk(&field.field_type, &mut max); + } + max + } +} + #[derive(Debug, Serialize, Deserialize, Clone, ToSchema)] #[serde(rename_all = "kebab-case")] pub struct NestedField { @@ -64,6 +175,8 @@ pub struct NestedField { pub required: bool, #[serde(rename = "type")] pub field_type: Type, + /// Omitted when absent rather than serialized as `"doc": null` (B14). + #[serde(default, skip_serializing_if = "Option::is_none")] pub doc: Option, } diff --git a/pangolin/pangolin_core/src/user.rs b/pangolin/pangolin_core/src/user.rs index 3c6eaac..2a88a5b 100644 --- a/pangolin/pangolin_core/src/user.rs +++ b/pangolin/pangolin_core/src/user.rs @@ -56,6 +56,14 @@ pub struct UserSession { pub role: UserRole, pub issued_at: DateTime, pub expires_at: DateTime, + /// The `jti` of the bearer token that produced this session, when there was + /// one. Revocation is keyed by `jti`, so logout has to revoke *this* id - + /// revoking `user_id` (as it once did) blacklists an id no token ever + /// carries, and the token keeps working until it expires naturally. + /// + /// `None` for sessions with no underlying JWT (API keys, root basic auth). + #[serde(default)] + pub token_id: Option, } /// Service user for API key authentication diff --git a/pangolin/pangolin_store/src/file_delete.rs b/pangolin/pangolin_store/src/file_delete.rs new file mode 100644 index 0000000..a2a59a0 --- /dev/null +++ b/pangolin/pangolin_store/src/file_delete.rs @@ -0,0 +1,64 @@ +//! Best-effort deletion of a single object at a warehouse location. +//! +//! Added for the metadata-orphan problem (B16d/B16g): the Iceberg commit loop +//! writes a full metadata file *before* attempting the compare-and-swap that +//! publishes it. On a lost CAS it retried and wrote a fresh file, abandoning the +//! previous one, and up to five orphans were left behind on a final give-up. +//! Orphaned metadata files are indistinguishable from live ones from the +//! outside, so they cannot be reaped later by inspection alone - they have to be +//! removed at the moment the writer knows they are unreferenced. +//! +//! Every backend routes here so the four of them cannot drift apart, which is +//! the failure mode most of the storage-layer audit findings share. + +use anyhow::Result; +use object_store::ObjectStore; +use std::collections::HashMap; + +/// Delete `location`, resolving credentials from `storage_config` when the +/// location points at object storage. +/// +/// A missing object is *not* an error: callers use this to clean up after a +/// failure, and the object may never have been written. +pub async fn delete_location( + storage_config: Option<&HashMap>, + location: &str, +) -> Result<()> { + if let Some(rest) = location.strip_prefix("file://") { + return match tokio::fs::remove_file(rest).await { + Ok(()) => Ok(()), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(e) => Err(anyhow::anyhow!("Failed to delete {}: {}", rest, e)), + }; + } + + let is_object_store = location.starts_with("s3://") + || location.starts_with("az://") + || location.starts_with("abfs://") + || location.starts_with("gs://"); + + if !is_object_store { + // A bare local path. + return match tokio::fs::remove_file(location).await { + Ok(()) => Ok(()), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(e) => Err(anyhow::anyhow!("Failed to delete {}: {}", location, e)), + }; + } + + let empty = HashMap::new(); + let config = storage_config.unwrap_or(&empty); + let store = crate::object_store_factory::create_object_store(config, location)?; + + let key = location + .split_once("://") + .and_then(|(_, rest)| rest.split_once('/')) + .map(|(_, key)| key) + .unwrap_or(location); + + match store.delete(&object_store::path::Path::from(key)).await { + Ok(()) => Ok(()), + Err(object_store::Error::NotFound { .. }) => Ok(()), + Err(e) => Err(anyhow::anyhow!("Failed to delete {}: {}", location, e)), + } +} diff --git a/pangolin/pangolin_store/src/lib.rs b/pangolin/pangolin_store/src/lib.rs index f645eba..aa6d1db 100644 --- a/pangolin/pangolin_store/src/lib.rs +++ b/pangolin/pangolin_store/src/lib.rs @@ -1,5 +1,6 @@ pub mod aws_utils; pub mod azure_signer; +pub mod file_delete; pub mod gcp_signer; pub mod memory; pub mod mongo; @@ -118,6 +119,7 @@ pub trait CatalogStore: Send + Sync + Signer { catalog_name: &str, namespace: Vec, ) -> Result<()>; + /// Merge `properties` into the namespace's existing properties. async fn update_namespace_properties( &self, tenant_id: Uuid, @@ -125,6 +127,19 @@ pub trait CatalogStore: Send + Sync + Signer { namespace: Vec, properties: std::collections::HashMap, ) -> Result<()>; + /// Replace the namespace's properties with `properties`. + /// + /// The merge-only method above cannot express a removal, which is why the + /// Iceberg `updateProperties` handler silently dropped every `removals` + /// entry while reporting success (B16h). Errors when the namespace does not + /// exist, so "updated nothing" and "no such namespace" are distinguishable. + async fn replace_namespace_properties( + &self, + tenant_id: Uuid, + catalog_name: &str, + namespace: Vec, + properties: std::collections::HashMap, + ) -> Result<()>; // Asset Operations async fn create_asset( @@ -340,6 +355,13 @@ pub trait CatalogStore: Send + Sync + Signer { // Generic File IO (for metadata files) async fn read_file(&self, location: &str) -> Result>; async fn write_file(&self, location: &str, content: Vec) -> Result<()>; + /// Delete a single file, resolving warehouse credentials from the location. + /// + /// Needed so the commit path can reclaim the metadata file it wrote before + /// losing a compare-and-swap (B16d), and so `create_table` can clean up + /// after a failed registration (B16g). Deleting something that is not there + /// succeeds. + async fn delete_file(&self, location: &str) -> Result<()>; // Maintenance Operations async fn expire_snapshots( diff --git a/pangolin/pangolin_store/src/memory/mod.rs b/pangolin/pangolin_store/src/memory/mod.rs index 7a95761..4d31a96 100644 --- a/pangolin/pangolin_store/src/memory/mod.rs +++ b/pangolin/pangolin_store/src/memory/mod.rs @@ -171,6 +171,17 @@ impl CatalogStore for MemoryStore { .await } + async fn replace_namespace_properties( + &self, + tenant_id: Uuid, + catalog_name: &str, + namespace: Vec, + properties: std::collections::HashMap, + ) -> Result<()> { + self.replace_namespace_properties_internal(tenant_id, catalog_name, namespace, properties) + .await + } + async fn count_namespaces(&self, tenant_id: Uuid) -> Result { self.count_namespaces_internal(tenant_id).await } @@ -413,6 +424,15 @@ impl CatalogStore for MemoryStore { self.write_file_internal(location, content).await } + async fn delete_file(&self, location: &str) -> Result<()> { + self.metadata_cache.invalidate(location).await; + self.files.remove(location); + let storage_config = self + .get_warehouse_for_location(location) + .map(|w| w.storage_config); + crate::file_delete::delete_location(storage_config.as_ref(), location).await + } + async fn expire_snapshots( &self, tenant_id: Uuid, diff --git a/pangolin/pangolin_store/src/memory/namespaces.rs b/pangolin/pangolin_store/src/memory/namespaces.rs index 643eef9..4233f6f 100644 --- a/pangolin/pangolin_store/src/memory/namespaces.rs +++ b/pangolin/pangolin_store/src/memory/namespaces.rs @@ -3,6 +3,17 @@ use anyhow::Result; use pangolin_core::model::*; use uuid::Uuid; +/// The single encoding of a namespace path into a map key. +/// +/// B17: create/get keyed with `join(".")` (via `Namespace::to_string`) while +/// delete/update keyed with `join("\x1F")`. The two never met, so a multi-level +/// namespace could never be deleted or updated on the memory backend - both +/// returned "Namespace not found" - while the SQL backends succeeded. One helper +/// used everywhere makes that class of divergence impossible. +fn ns_key(tenant_id: Uuid, catalog_name: &str, namespace: &[String]) -> (Uuid, String, String) { + (tenant_id, catalog_name.to_string(), namespace.join(".")) +} + impl MemoryStore { pub(crate) async fn create_namespace_internal( &self, @@ -10,7 +21,7 @@ impl MemoryStore { catalog_name: &str, namespace: Namespace, ) -> Result<()> { - let key = (tenant_id, catalog_name.to_string(), namespace.to_string()); + let key = ns_key(tenant_id, catalog_name, &namespace.name); self.namespaces.insert(key, namespace); Ok(()) } @@ -57,7 +68,7 @@ impl MemoryStore { catalog_name: &str, namespace: Vec, ) -> Result> { - let key = (tenant_id, catalog_name.to_string(), namespace.join(".")); + let key = ns_key(tenant_id, catalog_name, &namespace); if let Some(n) = self.namespaces.get(&key) { Ok(Some(n.value().clone())) } else { @@ -70,8 +81,7 @@ impl MemoryStore { catalog_name: &str, namespace: Vec, ) -> Result<()> { - let ns_str = namespace.join("\x1F"); - let key = (tenant_id, catalog_name.to_string(), ns_str); + let key = ns_key(tenant_id, catalog_name, &namespace); if self.namespaces.remove(&key).is_some() { Ok(()) } else { @@ -85,8 +95,7 @@ impl MemoryStore { namespace: Vec, properties: std::collections::HashMap, ) -> Result<()> { - let ns_str = namespace.join("\x1F"); - let key = (tenant_id, catalog_name.to_string(), ns_str); + let key = ns_key(tenant_id, catalog_name, &namespace); if let Some(mut ns) = self.namespaces.get_mut(&key) { ns.properties.extend(properties); @@ -95,6 +104,22 @@ impl MemoryStore { Err(anyhow::anyhow!("Namespace not found")) } } + pub(crate) async fn replace_namespace_properties_internal( + &self, + tenant_id: Uuid, + catalog_name: &str, + namespace: Vec, + properties: std::collections::HashMap, + ) -> Result<()> { + let key = ns_key(tenant_id, catalog_name, &namespace); + + if let Some(mut ns) = self.namespaces.get_mut(&key) { + ns.properties = properties; + Ok(()) + } else { + Err(anyhow::anyhow!("Namespace not found")) + } + } pub(crate) async fn count_namespaces_internal(&self, tenant_id: Uuid) -> Result { // Efficient counting for MemoryStore let count = self diff --git a/pangolin/pangolin_store/src/mongo/mod.rs b/pangolin/pangolin_store/src/mongo/mod.rs index 891b7d2..1fa18bc 100644 --- a/pangolin/pangolin_store/src/mongo/mod.rs +++ b/pangolin/pangolin_store/src/mongo/mod.rs @@ -154,6 +154,17 @@ impl CatalogStore for MongoStore { .await } + async fn replace_namespace_properties( + &self, + tenant_id: Uuid, + catalog_name: &str, + namespace: Vec, + properties: HashMap, + ) -> Result<()> { + self.replace_namespace_properties(tenant_id, catalog_name, namespace, properties) + .await + } + // Asset Operations async fn create_asset( &self, @@ -607,6 +618,14 @@ impl CatalogStore for MongoStore { async fn write_file(&self, path: &str, data: Vec) -> Result<()> { self.write_file(path, data).await } + async fn delete_file(&self, path: &str) -> Result<()> { + self.metadata_cache.invalidate(path).await; + let storage_config = self + .get_warehouse_for_location(path) + .await? + .map(|w| w.storage_config); + crate::file_delete::delete_location(storage_config.as_ref(), path).await + } // Access Requests async fn create_access_request(&self, request: AccessRequest) -> Result<()> { diff --git a/pangolin/pangolin_store/src/mongo/namespaces.rs b/pangolin/pangolin_store/src/mongo/namespaces.rs index 6c7d7c7..9cca7aa 100644 --- a/pangolin/pangolin_store/src/mongo/namespaces.rs +++ b/pangolin/pangolin_store/src/mongo/namespaces.rs @@ -120,6 +120,34 @@ impl MongoStore { Ok(()) } + /// Replace a namespace's properties wholesale; see the SQLite twin for why + /// a merge-only method could not implement Iceberg property removals (B16h). + pub async fn replace_namespace_properties( + &self, + tenant_id: Uuid, + catalog_name: &str, + namespace: Vec, + properties: HashMap, + ) -> Result<()> { + let filter = doc! { + "tenant_id": to_bson_uuid(tenant_id), + "catalog_name": catalog_name, + "name": namespace + }; + + let props = bson::to_bson(&properties)?; + let update = doc! { "$set": { "properties": props } }; + let result = self + .db + .collection::("namespaces") + .update_one(filter, update) + .await?; + if result.matched_count == 0 { + return Err(anyhow::anyhow!("Namespace not found")); + } + Ok(()) + } + pub async fn count_namespaces(&self, tenant_id: Uuid) -> Result { let filter = doc! { "tenant_id": to_bson_uuid(tenant_id) }; let count = self.namespaces().count_documents(filter).await?; diff --git a/pangolin/pangolin_store/src/postgres/main.rs b/pangolin/pangolin_store/src/postgres/main.rs index cf34019..dc6e05a 100644 --- a/pangolin/pangolin_store/src/postgres/main.rs +++ b/pangolin/pangolin_store/src/postgres/main.rs @@ -256,6 +256,17 @@ impl CatalogStore for PostgresStore { .await } + async fn replace_namespace_properties( + &self, + tenant_id: Uuid, + catalog_name: &str, + namespace: Vec, + properties: std::collections::HashMap, + ) -> Result<()> { + self.replace_namespace_properties(tenant_id, catalog_name, namespace, properties) + .await + } + // Asset Operations async fn create_asset( &self, @@ -1153,6 +1164,15 @@ impl CatalogStore for PostgresStore { } } + async fn delete_file(&self, path: &str) -> Result<()> { + self.metadata_cache.invalidate(path).await; + let storage_config = self + .get_warehouse_for_location(path) + .await? + .map(|w| w.storage_config); + crate::file_delete::delete_location(storage_config.as_ref(), path).await + } + // Tag Operations async fn create_tag(&self, tenant_id: Uuid, catalog_name: &str, tag: Tag) -> Result<()> { self.create_tag(tenant_id, catalog_name, tag).await diff --git a/pangolin/pangolin_store/src/postgres/namespaces.rs b/pangolin/pangolin_store/src/postgres/namespaces.rs index 22283ea..5f6956d 100644 --- a/pangolin/pangolin_store/src/postgres/namespaces.rs +++ b/pangolin/pangolin_store/src/postgres/namespaces.rs @@ -116,6 +116,28 @@ impl PostgresStore { Ok(()) } + /// Replace a namespace's properties wholesale; see the SQLite twin for why + /// a merge-only method could not implement Iceberg property removals (B16h). + pub async fn replace_namespace_properties( + &self, + tenant_id: Uuid, + catalog_name: &str, + namespace: Vec, + properties: std::collections::HashMap, + ) -> Result<()> { + let result = sqlx::query("UPDATE namespaces SET properties = $1 WHERE tenant_id = $2 AND catalog_name = $3 AND namespace_path = $4") + .bind(serde_json::to_value(&properties)?) + .bind(tenant_id) + .bind(catalog_name) + .bind(&namespace) + .execute(&self.pool) + .await?; + if result.rows_affected() == 0 { + return Err(anyhow::anyhow!("Namespace not found")); + } + Ok(()) + } + pub async fn count_namespaces(&self, tenant_id: Uuid) -> Result { let count: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM namespaces WHERE tenant_id = $1") .bind(tenant_id) diff --git a/pangolin/pangolin_store/src/sqlite/main.rs b/pangolin/pangolin_store/src/sqlite/main.rs index edf6375..a51b84b 100644 --- a/pangolin/pangolin_store/src/sqlite/main.rs +++ b/pangolin/pangolin_store/src/sqlite/main.rs @@ -327,6 +327,17 @@ impl CatalogStore for SqliteStore { .await } + async fn replace_namespace_properties( + &self, + tenant_id: Uuid, + catalog_name: &str, + namespace: Vec, + properties: std::collections::HashMap, + ) -> Result<()> { + self.replace_namespace_properties(tenant_id, catalog_name, namespace, properties) + .await + } + async fn create_asset( &self, tenant_id: Uuid, @@ -533,6 +544,14 @@ impl CatalogStore for SqliteStore { async fn write_file(&self, path: &str, data: Vec) -> Result<()> { self.write_file(path, data).await } + async fn delete_file(&self, path: &str) -> Result<()> { + self.metadata_cache.invalidate(path).await; + let storage_config = self + .get_warehouse_for_location(path) + .await? + .map(|w| w.storage_config); + crate::file_delete::delete_location(storage_config.as_ref(), path).await + } // Phase 3 & 4 (Access Control, Audit, Settings, etc) async fn create_user(&self, user: User) -> Result<()> { diff --git a/pangolin/pangolin_store/src/sqlite/namespaces.rs b/pangolin/pangolin_store/src/sqlite/namespaces.rs index d9f9d09..99714b2 100644 --- a/pangolin/pangolin_store/src/sqlite/namespaces.rs +++ b/pangolin/pangolin_store/src/sqlite/namespaces.rs @@ -120,4 +120,32 @@ impl SqliteStore { } Ok(()) } + + /// Replace a namespace's properties wholesale. + /// + /// `update_namespace_properties` merges, which cannot express a *removal*. + /// The Iceberg `updateProperties` endpoint takes both `updates` and + /// `removals`, and the handler used to ignore removals entirely while + /// reporting success (B16h); it now computes the resulting map and writes it + /// through here. + pub async fn replace_namespace_properties( + &self, + tenant_id: Uuid, + catalog_name: &str, + namespace: Vec, + properties: HashMap, + ) -> Result<()> { + let namespace_path = serde_json::to_string(&namespace)?; + let result = sqlx::query("UPDATE namespaces SET properties = ? WHERE tenant_id = ? AND catalog_name = ? AND namespace_path = ?") + .bind(serde_json::to_string(&properties)?) + .bind(tenant_id.to_string()) + .bind(catalog_name) + .bind(&namespace_path) + .execute(&self.pool) + .await?; + if result.rows_affected() == 0 { + return Err(anyhow::anyhow!("Namespace not found")); + } + Ok(()) + } } diff --git a/roadmap_aug10.md b/roadmap_aug10.md new file mode 100644 index 0000000..0ccf8b8 --- /dev/null +++ b/roadmap_aug10.md @@ -0,0 +1,423 @@ +# Pangolin — Roadmap & Audit Findings (August 10, 2026) + +**Scope:** full-repo audit — Rust workspace (`pangolin/`, 6 crates, ~58.5k LOC), Python SDK (`pypangolin/`), SvelteKit UI (`pangolin_ui/`), deployment assets, docs, and repo hygiene. +**Baseline:** the day after the 0.6.0 security/hardening release, which executed most of `AUDIT_EXECUTION_PLAN.md`. Items already fixed or explicitly documented as known limitations in the README are **not** re-reported here. + +--- + +## Overview + +The 0.6.0 release materially improved the project's substrate, and this audit verified those claims locally: + +- `cargo test --workspace --no-fail-fast` is **green** with no external services (verified today; the SQLite, memory, Mongo/Postgres-gated, and API integration targets all pass). +- `cargo fmt --all -- --check` is **clean**; `cargo clippy --workspace --all-targets` sits **exactly at its 36-warning budget** (`pangolin/clippy-warning-budget.txt`). +- CI (`.github/workflows/ci.yml`) now runs fmt, clippy + ratchet, tests (with and without live databases), `cargo audit`, helm lint/template, and a non-root Docker build. +- The Iceberg commit path (`pangolin_api/src/iceberg/commit.rs`) is now a well-tested pure module enforcing all documented requirements and updates. + +The remaining problems cluster in five places: + +1. **A new class of API-layer authorization bypasses (`pangolin_api`).** Several management and Iceberg handlers are mounted behind authentication but perform **no authorization check at all** — most severely, any authenticated principal can mint a `Root` JWT for any tenant, and any tenant member can vend read+write cloud credentials for a whole warehouse. These are as serious as the OAuth issues 0.6.0 fixed and were *not* in that release's scope. **This is the highest-priority cluster.** +2. **Backend parity and correctness in `pangolin_store`** — the four backends (memory, SQLite, Postgres, Mongo) disagree on tenant scoping, branch scoping, serde formats, CAS enforcement, and pagination determinism. Several are severe (a cross-tenant audit read, a revocation no-op on Mongo, a panic in Postgres search). +3. **Iceberg metadata JSON conformance and commit-path edge cases in `pangolin_core`/`pangolin_api`** — field naming and missing required fields mean metadata files Pangolin writes are not readable as spec-conformant v2 metadata by external engines, and several commit-path handlers mis-handle branches, nested namespaces, and client-supplied values. +4. **The UI's integration seams** — env-var mismatches, endpoints that don't exist on the server, missing 401 handling, and dead tenant switching. +5. **Client ↔ server contract drift** — the Python SDK and both Rust CLIs call dozens of wrong endpoints / wrong field names that fail silently or 404/422, because nothing tests them against the real router. Plus packaging/deployment drift: the quick-start `docker compose up` cannot start the API, and the release compose pins 0.2.0. + +The Iceberg *commit-application* module (`commit.rs`) is genuinely solid and well-tested. The rest of the API crate, despite compiling cleanly and passing CI, carries real authorization gaps that CI cannot see because there are no authz tests. Most fixes below are contained and file-level; the meta-fix is a permission-matrix test plus a client↔server contract test. + +--- + +## Bugs + +Ordered by severity. Every location is verified against the working tree as of 2026-08-10. The `pangolin_api` authorization items below were spot-verified by reading the handler signatures directly (confirmed: the cited handlers take no `UserSession` and/or contain no `check_permission` call). + +### Critical — API authorization bypasses (`pangolin_api`) + +These handlers all sit behind `auth_middleware` (so they need a valid credential) but perform no authorization, so the bar is "any authenticated principal — including a lowest-privilege `TenantUser` or any service-user API key." + +**B0a. `POST /api/v1/tokens` mints arbitrary-role, arbitrary-tenant JWTs for any authenticated caller — full privilege escalation.** +- `pangolin/pangolin_api/src/token_handlers.rs:45-113` (route `lib.rs:506`). **Verified:** `generate_token` takes only `State` + `Json` — no `Extension`, no permission check — and maps a body-supplied `roles: ["Root"]` straight into signed `Claims` (lines 70-104). Any `TenantUser` can POST `{"tenant_id":"","roles":["Root"]}` and receive a valid `Root` token; `check_permission` short-circuits `Ok(true)` for `Root` (`authz.rs:16-18`). +- **Fix:** require `Extension`; reject unless caller is `Root`, or is `TenantAdmin` with `payload.tenant_id == session.tenant_id` and a role not exceeding the caller's. Never trust `roles` from the body for a non-Root caller. + +**B0b. Credential-vending endpoint has no authz and never checks the table — any tenant member gets read+write warehouse credentials.** +- `pangolin/pangolin_api/src/signing_handlers.rs:200-250`. **Verified:** `get_table_credentials` takes no `UserSession`, calls no `check_permission`, hardcodes `permissions = ["read","write"]` (line 243), and never looks up the asset (`namespace`/`table` are only string-concatenated into the resource path). Any authenticated tenant member obtains read+write cloud storage credentials for the entire warehouse, for a table they have no rights to and that need not exist. Highest-value endpoint in the API to gate. +- **Fix:** load the asset; `check_permission(Read)` for read-only vending, `Write` before adding `"write"`; derive `permissions` from the caller's actual grants. + +**B0c. `rename_table` has no permission check.** +- `pangolin/pangolin_api/src/iceberg/tables.rs:992-1036`. **Verified:** the handler binds `Extension(session)` but never calls `check_permission` (every sibling table handler does). Any tenant member can move any table into any namespace — an effective delete/DoS and a way to smuggle a table into a namespace where they have read rights. +- **Fix:** `check_permission(Write|Delete, source_asset)` and `check_permission(Create, dest_namespace)` before `rename_asset`. + +**B0d. `update_namespace_properties` has no permission check.** +- `pangolin/pangolin_api/src/iceberg/namespaces.rs:307-363` — binds `Extension(_session)` (deliberately discarded) and never checks. Any tenant member can rewrite namespace properties including `location`, which later table creation derives paths from. This handler also never resolves the catalog (needs a `get_catalog` + 404). +- **Fix:** resolve the catalog, then `check_permission(Write, Namespace{...})`. + +**B0e. View endpoints have no permission checks.** +- `pangolin/pangolin_api/src/asset_handlers.rs:66-109` (`create_view`), `:128-158` (`get_view`). Neither takes a `UserSession` or checks. Any tenant member can create views in any namespace and read any view's SQL text (`properties["sql"]`). +- **Fix:** mirror `create_table`/`load_table` scope checks. + +**B0f. `perform_maintenance`: no authz AND hardcoded `"default"` catalog.** +- `pangolin/pangolin_api/src/iceberg/tables.rs:146-198`. Two bugs: (1) the catalog from the path (`_prefix`) is discarded and the literal `"default"` is passed to `expire_snapshots`/`remove_orphan_files` (lines 166, 184) — destructive maintenance runs against the wrong catalog; (2) no `UserSession`/`check_permission` — any tenant member can trigger snapshot expiry and orphan-file deletion on tables. +- **Fix:** use `_prefix` as the catalog; add `Write`/`Delete` checks. + +**B0g. Iceberg OAuth token endpoint ignores service-user expiry.** +- `pangolin/pangolin_api/src/iceberg/oauth.rs:88-92`. **Verified:** checks only `service_user.active`, not `is_valid()` (`= active && !is_expired()`, `pangolin_core/src/user.rs:126-128`) — the API-key path uses `is_valid()` correctly (`auth_middleware.rs:233`). An **expired** service user can still exchange `client_credentials` for a fresh 1-hour JWT, fully bypassing key expiry. (Secondary: this public endpoint runs an unthrottled bcrypt per call, keyed by attacker-supplied UUID.) +- **Fix:** `if !service_user.is_valid() { return unauthorized }`. + +**B0h. `PANGOLIN_DEV_MODE=true` waives the NO_AUTH public-bind guard.** +- `pangolin/pangolin_api/src/config.rs:216-218`. **Verified:** `if no_auth && !dev_mode && !is_loopback(...)`. The `!dev_mode` term means `PANGOLIN_NO_AUTH=true PANGOLIN_DEV_MODE=true` on the default `0.0.0.0` bind starts happily and treats every anonymous request as `TenantAdmin` (`auth_middleware.rs:209-221`) — and both flags are commonly set together in compose/dev setups. This breaks the invariant `auth_middleware.rs:203-204` documents. +- **Fix:** drop `!dev_mode`; the bind restriction should be unconditional whenever `no_auth` is on. + +**B0i. `PermissionScope::Tenant` matches unconditionally — cross-tenant grant leak.** +- `pangolin/pangolin_api/src/authz_utils.rs:21,52,92`: `PermissionScope::Tenant => true` in all three access checks, never comparing `perm.tenant_id` against the resource's tenant. A `Tenant`-scoped grant issued in tenant A satisfies access for resources in tenant B. Masked today only because callers pre-scope at the store layer; any cross-tenant result path (root impersonation, search, dashboards) leaks. +- **Fix:** thread the resource tenant id in and require `perm.tenant_id == resource_tenant_id`. + +**B0j. Logout / `revoke_current_token` revokes the wrong ID — tokens are never actually revoked.** +- `pangolin/pangolin_api/src/token_handlers.rs:196-201`. **Verified:** revokes `session.user_id`, but the middleware checks revocation by the token's `jti` (`auth_middleware.rs:372-378`), a fresh `Uuid::new_v4()` per token. Logout returns 200 and the token keeps working until natural expiry (24h). Wrong-variable bug; also blocks `rotate_token`. +- **Fix:** carry `jti` into `UserSession` and revoke that. + +**B0k. `POST /api/v1/oauth/exchange` is not in the public-path allowlist — the OAuth login flow cannot complete.** +- Route `lib.rs:516-519`, handler `oauth_handlers.rs:339`, allowlist `public_paths.rs:27-56`. **Verified:** `is_public_path` does not match `["api","v1","oauth","exchange"]`, so the middleware demands a bearer token on the very endpoint whose job is to obtain the first token. The 0.6.0 A-8 remediation (callback → one-time code → POST exchange) is unreachable in production — the browser lands with `?code=...` it can never redeem. +- **Fix:** add `["api","v1","oauth","exchange"] => true` and a regression test. + +**B0l. OAuth account linking by unverified email.** +- `pangolin/pangolin_api/src/oauth_handlers.rs:193-198`: existing-user match includes `|| u.email == user_info.email` with no `email_verified` check and no provider/domain binding. An attacker who sets a matching email on any configured provider (GitHub allows unverified addresses) logs in as that Pangolin user, including the seeded `TenantAdmin`. (Also O(all users) per login; truncated pagination silently forces a `create_user` instead.) +- **Fix:** match on `(provider, subject)` only; gate email linking behind a verified `id_token` + operator domain allowlist. + +**B0m. Request-controlled panics in token issuance.** +- `pangolin/pangolin_api/src/token_handlers.rs:56-61` (also `:143-145`): `chrono::Duration::hours(expires_in as i64)` panics for a huge `expires_in_hours` before the `.unwrap()` on `checked_add_signed` even runs. `expires_in_hours` is attacker-controlled `Option`; there is no `CatchPanicLayer` in the stack (`lib.rs:551-575`), so the panic aborts the connection task. +- **Fix:** clamp `expires_in` to a configured max, use `checked_*` and return 400; add `tower_http::catch_panic::CatchPanicLayer`. + +### Critical — data integrity & tenant isolation (storage layer) + +**B1. Mongo `get_audit_event` ignores `tenant_id` — cross-tenant audit-log read.** +- `pangolin/pangolin_store/src/mongo/mod.rs:461-467` discards the `tenant_id` parameter (`_tenant_id`), and `pangolin/pangolin_store/src/mongo/audit.rs:34-42` filters only on `{ "id": ... }`. +- Postgres (`postgres/audit.rs:164`) and SQLite (`sqlite/audit_logs.rs:176`) both scope by tenant. On Mongo, any tenant holding an audit-event UUID can read another tenant's audit record (username, IP, resource names, metadata). +- **Fix:** add `"tenant_id": to_bson_uuid(tenant_id)` to the filter and thread the parameter through. + +**B2. Mongo token revocation is a silent no-op — revoked JWTs stay valid.** +- `pangolin/pangolin_store/src/mongo/tokens.rs:79-101`: revocations are inserted via serde (UUID → string), but `is_token_revoked` queries with a BSON Binary UUID (`to_bson_uuid`). The filter can never match, so revocation checks always return `false` on the Mongo backend. +- `cleanup_expired_tokens` (`tokens.rs:103-112`) has the same type mismatch (`$lt` BSON DateTime vs stored RFC3339 string) — cleanup never deletes anything and the collection grows unbounded. +- **Fix:** write via an explicit `doc!` using `to_bson_uuid`/`Bson::DateTime` (mirroring `store_token` at lines 66-74). Add a store-level revoke→check roundtrip test that runs against all backends. + +**B3. SQLite `delete_branch` references a non-existent column and is non-transactional — orphans branch assets.** +- `pangolin/pangolin_store/src/sqlite/branches.rs:109-130`: the branch row is deleted and committed, then `DELETE FROM assets ... AND branch = ?` fails ("no such column" — the schema column is `branch_name`, `sql/sqlite_schema.sql:71`). The branch is gone, its assets are permanently orphaned, and the caller receives an error. +- Postgres was fixed for exactly this (`postgres/branches.rs:127-129` comment) and wraps both statements in a transaction; SQLite was never patched. +- **Fix:** `branch = ?` → `branch_name = ?`, and wrap both statements in `self.pool.begin()`/`tx.commit()`. + +**B4. Postgres `search_assets` panics on any match — `TEXT[]` decoded as `String`.** +- `pangolin/pangolin_store/src/postgres/main.rs:1391-1392`: `row.get::("namespace_path")` against a `TEXT[]` column (`migrations/20251212000000_initial_schema.sql:47`). `sqlx::Row::get` panics on decode failure, so any search with ≥1 hit panics the request. The correct decode (`Vec`) is used at `main.rs:1449` and `postgres/assets.rs:87`. +- SQLite has the sibling bug at `sqlite/business_metadata.rs:156-157` — no panic, but every result's namespace is one element of raw JSON (`["[\"a\",\"b\"]"]`) instead of the parsed path. +- **Fix (pg):** `let namespace: Vec = row.get("namespace_path");` **Fix (sqlite):** `serde_json::from_str(&namespace_path).unwrap_or_default()`. + +**B5. Mongo `update_metadata_location` drops the compare-and-swap — lost Iceberg commits.** +- `pangolin/pangolin_store/src/mongo/main.rs:213`: `_expected_location` is ignored; the update is an unconditional `$set`. Memory (`memory/io.rs:62`), Postgres (`postgres/main.rs:1552,1564`), and SQLite (`sqlite/assets.rs:383`) all enforce the CAS. On Mongo, two concurrent commits both "succeed" and one snapshot is silently lost — the exact failure class 0.6.0 fixed at the API layer. +- **Fix:** put the expected location in the update filter (`"properties.metadata_location": expected`, or `$exists: false` when `None`) and error when `modified_count == 0`. This needs no multi-document transaction, so it works on standalone `mongod`. + +**B6. Memory `delete_catalog` corrupts other tenants' asset-by-id index.** +- `pangolin/pangolin_store/src/memory/catalogs.rs:124`: `self.assets_by_id.retain(|_, v| v.0 != name)` filters on catalog **name only** — tenant A deleting catalog `sales` breaks `get_asset_by_id` for tenant B's `sales`. +- **Fix:** include `tenant_id` in the `assets_by_id` value and match on both. + +**B7. All three persistent backends silently rewrite asset types to `IcebergTable`.** +- Write path stores `format!("{:?}", asset.kind)`, read path parses only `IcebergTable`/`View` and defaults everything else: `postgres/assets.rs:25` + `:53-57,:89-93,:140-144`; `sqlite/assets.rs:27` + `:142-146,:181-185,:234-238`; `mongo/assets.rs:36` + `:75-79,:131-135,:183-187`. +- `AssetType` has 17 variants (`pangolin_core/src/model.rs:93-111`) — a `DeltaTable`, `MlModel`, `Lance`, etc. round-trips as `IcebergTable`. This defeats a headline feature ("tracks any lakehouse asset type"). +- **Fix:** serialize via serde (the enum already has a rename policy) and hard-error on unknown values; add a parity test that round-trips a third variant. + +### Critical — deployment & quick start + +**B8. `docker compose up` cannot start the API.** +- `docker-compose.yml:34-49` (service `pangolin-api`) sets no `PANGOLIN_JWT_SECRET`; since 0.6.0 the server refuses to start without one (`pangolin/pangolin_api/src/config.rs:49`), and `PANGOLIN_NO_AUTH` is refused on the default `0.0.0.0` bind (`config.rs:215-217`). The documented quick start yields a crash-looping container. +- **Fix:** add `PANGOLIN_JWT_SECRET=${PANGOLIN_JWT_SECRET:?generate with openssl rand -base64 48}` to the compose file (fail fast with a clear message), and document it in the quick start. + +**B9. Compose files set the wrong storage env var.** +- `docker-compose.yml:43` and `docker-compose.release.yml:42` set `PANGOLIN_STORE_TYPE`; the server reads `PANGOLIN_STORAGE_TYPE` (`pangolin/pangolin_api/src/main.rs:187`). The variable is silently ignored — anyone editing it to `postgres` would still get the memory backend. +- **Fix:** rename to `PANGOLIN_STORAGE_TYPE` in both files (and see B18 for the docs side). + +**B10. `docker-compose.release.yml` is stale and self-broken.** +- Line 37 pins `image: alexmerced/pangolin-api:0.2.0` (four releases behind the 0.6.0 workspace), and line 68 runs `scripts/test_release_v0.2.0.py`, which does not exist in `scripts/`. +- **Fix:** pin `0.6.0` (or parameterize `${PANGOLIN_VERSION}`), point at a real verification script (e.g. `scripts/integration_test.py`), and add a CI job that `docker compose -f docker-compose.release.yml config`-validates the file. + +### High — Iceberg spec conformance (core metadata model) + +**B11. Table metadata JSON is not spec-conformant: wrong name for `default-spec-id`.** +- `pangolin/pangolin_core/src/iceberg_metadata.rs:17`: `current_partition_spec_id` under `rename_all = "kebab-case"` serializes as `current-partition-spec-id`. The Iceberg v2 spec field is `default-spec-id`. Metadata files Pangolin writes are unreadable as conformant v2 metadata by external engines reading the file directly, and a conformant engine's metadata can't round-trip in. +- **Fix:** `#[serde(rename = "default-spec-id", alias = "current-partition-spec-id")]` so old Pangolin-written files still parse. + +**B12. Required v2 field `last-partition-id` is missing entirely.** +- `pangolin/pangolin_core/src/iceberg_metadata.rs:8-33` (struct `TableMetadata`); `grep -r last_partition_id` over the workspace returns nothing. The v2 spec requires `last-partition-id` (highest assigned partition field id); Java-based readers reject metadata without it. +- **Fix:** add the field with a serde default computed from `partition_specs` on read; maintain it in `apply_updates`' `AddSpec` arm (`pangolin_api/src/iceberg/commit.rs:369-375`). + +**B13. `metadata-log` is initialized empty and never appended.** +- `pangolin/pangolin_api/src/iceberg/tables.rs:400` creates `metadata_log: Some(vec![])`; no code path ever pushes a `MetadataLogEntry`. The spec expects each commit to record the previous metadata file — engines use it for metadata time-travel and previous-metadata cleanup (`write.metadata.previous-versions-max`). +- **Fix:** in the commit handler, before writing the new metadata file, append `{timestamp_ms, metadata_file: }` and truncate to the configured max. + +**B14. `Schema` JSON omits `"type": "struct"`; optional fields serialize as explicit `null`.** +- `pangolin/pangolin_core/src/iceberg_metadata.rs:53` (struct `Schema`) has no `type` field — spec schemas are struct types and conformant writers emit `"type": "struct"`. Also `NestedField.doc` and `Schema.identifier_field_ids` lack `skip_serializing_if`, producing `"doc": null` noise that some strict parsers reject. +- **Fix:** add a `#[serde(rename = "type")] type_: String` defaulted to `"struct"`, and `skip_serializing_if = "Option::is_none"` on the optional fields. + +**B15. Client-supplied sequence numbers can jump the counter arbitrarily.** +- `pangolin/pangolin_api/src/iceberg/commit.rs:481-485`: a snapshot whose `sequence-number` exceeds `last_sequence_number` is honored verbatim. A client can submit `i64::MAX`, after which the next commit computes `last_sequence_number + 1` (line 480) — overflow (panic in debug builds, wrap in release, corrupt ordering either way). +- **Fix:** accept the client value only if it equals `last_sequence_number + 1`; otherwise assign the next counter value (or reject with `CommitError::Invalid`). + +**B16. Committing to a non-main branch mutates main-visible state.** +- `pangolin/pangolin_api/src/iceberg/commit.rs:497` sets `current_snapshot_id` to the new snapshot for **any** branch, and lines 518-529 fabricate a `main` ref pointing at the branch's snapshot when no `main` ref exists. If a metadata document is ever shared across Pangolin branches (or exported), a `dev`-branch commit changes what `main` readers resolve. +- **Fix:** only update `current_snapshot_id`/insert the `main` ref when `branch == MAIN_REF`; add a test asserting a feature-branch commit leaves `ref_snapshot_id(metadata, "main")` unchanged. + +### High — additional Iceberg commit/handler bugs (`pangolin_api`) + +**B16a. Nested namespaces are parsed two different ways — commit/delete/HEAD on a nested namespace 404.** +- In `pangolin/pangolin_api/src/iceberg/tables.rs`, `list_tables`/`create_table`/`load_table` use `parse_table_identifier` (lines 75, 266, 570) yielding a single-element namespace, while `update_table`/`delete_table`/`table_exists` use `namespace.split('\x1F')` (lines 747, 1118, 1226) yielding a multi-element path. Creating a table in namespace `a\x1Fb` registers it under `["a\x1Fb"]`, but the commit path looks it up under `["a","b"]` → `404 Table not found`, and the CAS loop never runs. `parse_table_identifier` also splits on `@`, so `update_table` mishandles a `ns@branch` suffix that `load_table` accepts. +- **Fix:** one shared `parse_namespace(&str) -> (Vec, Option)` (split on `0x1F`, strip trailing `@branch`) used in all six handlers plus `namespaces.rs:242,334` and `asset_handlers.rs:79,139`. + +**B16b. `last-updated-ms` only changes on `add-snapshot` commits.** +- `pangolin/pangolin_api/src/iceberg/commit.rs:496` is the only assignment (inside `add_snapshot`). A commit of only `set-properties`/`add-schema`/`set-location`/`add-spec`/`set-snapshot-ref`/`remove-snapshots` publishes a new metadata file with an unchanged `last-updated-ms`, so consumers that order or dedupe metadata by that field treat the two versions as identical. +- **Fix:** set `metadata.last_updated_ms = Utc::now().timestamp_millis()` once in `update_table` after `apply_updates` returns `Ok`. + +**B16c. `set-current-schema: -1` (and `set-default-spec`/`set-default-sort-order: -1`) resolve against the full list, not "added in this commit."** +- `pangolin/pangolin_api/src/iceberg/commit.rs:275-295, 377-395, 407-425`. Because `metadata.schemas` is never empty for an existing table, a `-1` sent *without* a preceding `add-schema` doesn't hit the intended `None`/error arm — it silently points the table at whatever is `.last()` in the persisted vector (arbitrary if not in creation order). The error message asserts a check the code doesn't perform. `Add*` arms also don't reject duplicate IDs. +- **Fix:** track `last_added_{schema,spec,sort_order}_id: Option` locals in `apply_updates`, resolve `-1` to those, error when `None`; reject or remap duplicate `Add*` IDs. + +**B16d. Metadata files are written before the CAS and never reclaimed on loss — object-storage leak under contention.** +- `pangolin/pangolin_api/src/iceberg/tables.rs:857-944`: each of up-to-5 retry iterations writes a full metadata file then attempts `update_metadata_location`; on CAS loss it `continue`s, orphaning the just-written file, and up to 5 orphans remain on final give-up. No reaper; orphans are indistinguishable from live metadata (compounds B13's empty `metadata-log`). +- **Fix:** best-effort delete `new_metadata_location` on the CAS-failure branch, or derive the location deterministically from `(table_uuid, version)` so a retry overwrites. + +**B16e. `create_table` returns the table *directory* as `metadata-location`.** +- `pangolin/pangolin_api/src/iceberg/tables.rs:496` returns `Some(location.clone())` (the table root) where it should return `metadata_location` (the file, computed at line 405). `load_table` returns the file correctly (line 678). A client keeping the returned `Table` (PyIceberg does) has a `metadata_location` it cannot read or refresh from. Straight wrong-variable bug. +- **Fix:** `Some(metadata_location.clone())`. + +**B16f. `create_table` hand-parses the schema: drops nullability, widens `int`→`long`, and silently deletes complex-typed columns.** +- `pangolin/pangolin_api/src/iceberg/tables.rs:331-371`: `required = false` hardcoded (line 338); `"int"|"integer" => long` (line 342); `field.get("type")?.as_str()?` inside a `filter_map` returns `None` for any `struct`/`list`/`map`/`decimal`/`fixed` field, so those columns are dropped and `last_column_id` is miscomputed — table creation succeeds `200 OK` with a schema missing columns. +- **Fix:** deserialize the incoming schema straight into `pangolin_core::iceberg_metadata::Schema` (as `commit.rs:265` does) and 400 on failure. + +**B16g. `create_table` registers the asset before the metadata file is durable.** +- `pangolin/pangolin_api/src/iceberg/tables.rs:426-441`: `create_asset` runs first, `write_file` second; if the write fails the asset remains registered pointing at a nonexistent file, permanently breaking the table (`load_table` 500s, `update_table` 404s) with no repair path. Inverted vs. the commit path. +- **Fix:** write the metadata file first, then `create_asset`; best-effort delete the file on `create_asset` failure. + +**B16h. `update_namespace_properties` silently ignores `removals`.** +- `pangolin/pangolin_api/src/iceberg/namespaces.rs:336-347`: only `updates` are applied; a request with `removals` gets `200 OK` with `removed: []`/`missing: []` while nothing was removed — the exact "silent success" failure class 0.6.0 fixed elsewhere. +- **Fix:** implement removals and populate `updated`/`removed`/`missing` honestly. + +**B16i. Iceberg list pagination is non-conforming; `next-page-token` is never returned.** +- `pangolin/pangolin_api/src/iceberg/types.rs:12-15,107-110`, `namespaces.rs:40,93`, `tables.rs:46,102`: the API uses `{limit,offset}` and the list responses have no `next-page-token` field. A spec client sending `pageToken`/`pageSize` has both ignored and cannot detect truncation → silent partial results on large namespaces. +- **Fix:** accept `pageToken`/`pageSize`, encode offset into an opaque token, add `next-page-token` to both response types. + +**B16j. Most Iceberg handlers still return plain-text bodies, not the spec error envelope.** +- `iceberg/tables.rs:69,84,259,276,586,600,739,760,774,801,813,1110,1131,1145` and `iceberg/namespaces.rs:70,82,146,158,236,252,285,350` return `(StatusCode, "…")` text bodies; only `commit_error_response` uses the `iceberg/error.rs` helpers. Engines that switch on `error.type` to decide commit-retry cannot parse these. (This is the old A-6 — it was **not** in the 0.6.0 fix set despite the module doc reading as though resolved.) +- **Fix:** route every bare-tuple return in `iceberg/` through the `iceberg_error(...)` helpers. + +**B16k. Federated forwarding is missing on create/delete namespace and the namespace tree.** +- `pangolin/pangolin_api/src/iceberg/namespaces.rs:127-205, 224-287, 381-464` don't call `check_and_forward_if_federated` (which `list_namespaces`/`update_namespace_properties` do). On a `Federated` catalog, creating a namespace makes a local shadow returning `200` while `GET` lists the remote — the two views permanently diverge; delete reports success for a namespace still present upstream. +- **Fix:** add the forwarding guard to all three; audit `asset_handlers`/`signing_handlers` for the same omission. + +### High — API middleware / reliability (`pangolin_api`) + +**B16l. Timeout layer is nested inside the concurrency limiter — queued requests have no deadline.** +- `pangolin/pangolin_api/src/lib.rs:561-569`: effective order is `concurrency-limit → timeout → body-limit`, so a request waiting for one of the 512 permits has no deadline; the 30s timer starts only after admission. Under sustained overload the queue grows unbounded and clients see latency far past `PANGOLIN_REQUEST_TIMEOUT_SECS`. +- **Fix:** register `TimeoutLayer` after (outside) `GlobalConcurrencyLimitLayer`; consider `LoadShedLayer` outside both to return 503. + +**B16m. `delete_warehouse` invalidates the cache *before* deleting — a concurrent read re-poisons it.** +- `pangolin/pangolin_api/src/cached_store.rs:110-113`: invalidate then delete. A `get_warehouse` racing in between misses, reads the still-present row, and re-inserts it with a full TTL — the deleted warehouse's cloud credentials keep being vended for up to the cache TTL after delete returns success. +- **Fix:** delete first, then invalidate (and invalidate again on the error path). + +**B16n. `shutdown_grace` is logged but never applied — SIGTERM can hang forever.** +- `pangolin/pangolin_api/src/main.rs:340-370`: the grace value is only logged; there is no `sleep` and no `timeout` around the drain, so `with_graceful_shutdown` waits for in-flight connections indefinitely. A single hung upstream blocks SIGTERM past the k8s grace period into SIGKILL mid-commit; `PANGOLIN_SHUTDOWN_GRACE_SECS` has no effect. +- **Fix:** sleep briefly for LB deregistration, then wrap the serve future in `tokio::time::timeout(grace, …)`. + +**B16o. Token revocation is skipped for tokens with a missing/malformed `jti`; API-key auth runs before the public-path check.** +- `auth_middleware.rs:372-393`: both `if let`s fail open, so a token minted without a UUID `jti` is unrevocable for its lifetime (`Claims.jti` is `Option` "for compatibility"). Separately, the `X-API-Key` branch (`auth_middleware.rs:227-277`) returns before `is_public_path` (line 275), so a client sending `X-API-Key` globally can't reach `/v1/config`, `/health`, or the OAuth token endpoint — the opposite of the documented ordering. +- **Fix:** reject a present-but-unparseable `jti`; gate legacy no-`jti` acceptance behind a default-off flag; move the public-path check above the API-key branch. + +### High — storage-layer parity & correctness (continued) + +**B17. Memory backend namespace delete/update use a different key encoding than create — always "not found" for nested namespaces.** +- `pangolin/pangolin_store/src/memory/namespaces.rs`: create/get key with `join(".")` (lines 13, 60) but delete/update key with `join("\x1F")` (lines 73, 88). Multi-level namespaces can never be deleted or updated on the memory backend, while the SQL backends succeed. +- **Fix:** a single shared `ns_key()` helper using `join(".")`. + +**B18. SQLite `PRAGMA foreign_keys = ON` applies to one pooled connection.** +- `pangolin/pangolin_store/src/sqlite/main.rs:58,118,158`: the pragma is per-connection; `execute(&pool)` configures one arbitrary connection out of the pool, so `ON DELETE CASCADE` fires nondeterministically depending on which connection serves a request. Line 118's `OFF` can additionally stick on a connection the later `ON` never touches. +- **Fix:** `SqliteConnectOptions::new().foreign_keys(true)` (or an `after_connect` hook) so every connection is configured. + +**B19. SQLite `get_metadata_location` ignores the branch — cross-branch metadata reads.** +- `pangolin/pangolin_store/src/sqlite/assets.rs:322,327`: `_branch` is discarded and the query matches rows from every branch; `fetch_optional` returns an arbitrary one. Reading a table on `dev` can return `main`'s pointer. Postgres (`postgres/main.rs:1521`) and Mongo (`mongo/main.rs:196`) scope by branch. +- **Fix:** `AND branch_name = ?` binding `branch.unwrap_or("main")`. + +**B20. SQLite `update_metadata_location` leaves the `metadata_location` column stale.** +- `pangolin/pangolin_store/src/sqlite/assets.rs:393` updates only `properties`, but reads populate `Asset.location` from the column (`assets.rs:152-154,244-246`) — so on SQLite, `Asset.location` is frozen at creation time after every Iceberg commit. Postgres updates both (`postgres/main.rs:1552`). +- **Fix:** `UPDATE assets SET metadata_location = ?, properties = ? ...`. + +**B21. SQLite `delete_catalog`: non-transactional cascade that purges children even when the catalog doesn't exist.** +- `pangolin/pangolin_store/src/sqlite/catalogs.rs:146-189`: five sequential deletes with no transaction, and the "not found" check is the **last** statement — `delete_catalog(tenant, "nonexistent")` deletes matching tags/branches/assets/namespaces, then errors. Postgres wraps the identical cascade in a transaction (`postgres/catalogs.rs:158-204`). +- **Fix:** check existence first, then run the cascade inside `pool.begin()`/`tx.commit()`, matching Postgres. + +**B22. SQLite audit log: multi-word actions all read back as `CreateCatalog`.** +- `pangolin/pangolin_store/src/sqlite/audit_logs.rs:22` stores `format!("{:?}", action)` (`"CreateBranch"`), and lines 107-108/187-188 deserialize `"createbranch"` against serde's `snake_case` (`create_branch`) — no match — then `unwrap_or(AuditAction::CreateCatalog)` swallows it. The audit trail on SQLite misattributes nearly every action. +- **Fix:** persist the serde name symmetrically and propagate parse errors instead of defaulting. + +**B23. Mongo audit filters never match; listing is unsorted and unbounded.** +- `pangolin/pangolin_store/src/mongo/audit.rs:80-85` filters with Debug names (`"CreateBranch"`) against serde-written snake_case documents — action/resource-type filters always return zero rows and `count_audit_events` returns 0. `resource_id`/`result` filters are ignored, and `list_audit_events` (lines 58-71) applies no sort/limit/offset while the SQL backends use `ORDER BY timestamp DESC LIMIT 100`. +- **Fix:** build filters with `bson::to_bson`, add the missing fields, and mirror the SQL sort/limit/skip. + +**B24. Postgres `list_catalogs` fabricates `catalog_type: Local` and `federated_config: None`.** +- `pangolin/pangolin_store/src/postgres/catalogs.rs:65,77,80`: the SELECT omits both columns and hardcodes the values; `get_catalog` decodes them properly. Every federated catalog appears Local in a Postgres listing — anything branching on `catalog_type` over a listing takes the wrong path. SQLite and Mongo return real values. +- **Fix:** select and decode `catalog_type, federated_config` as in `get_catalog` (line 25). + +**B25. Memory `merge_branch` reuses asset IDs and never advances the target head.** +- `pangolin/pangolin_store/src/memory/branches.rs:161-212`: copied assets keep the same `asset.id`, so `assets_by_id` is repointed at the target-branch copy (source lookups now resolve wrong); and `target_branch.head_commit_id` is never set, unlike all three other backends (`postgres/branches.rs:180`, `sqlite/branches.rs:161`, `mongo/branches.rs:172-176`). +- **Fix:** mint `Uuid::new_v4()` per copied asset and set the target head from the source branch. + +**B26. Memory CAS skips the check entirely when `expected_location` is `None`.** +- `pangolin/pangolin_store/src/memory/io.rs:62`: `if let Some(expected) = ...` means the create-path CAS (expected = None must require "no existing location") is not enforced — a create-table race that Postgres/SQLite correctly reject silently succeeds in dev/tests. +- **Fix:** compare `current_loc != expected_location` unconditionally (the SQLite form). + +**B27. Pagination is nondeterministic almost everywhere: `LIMIT/OFFSET` with no `ORDER BY`.** +- Only three list queries in the crate are ordered (`postgres/assets.rs:127`, `postgres/service_users.rs:77`, `postgres/access_requests.rs:62`). Every other paginated query in postgres/sqlite/mongo, and every memory-backend `skip()/take()` over DashMap iteration order, can repeat or skip rows between pages. (Full site list: see the storage audit notes — ~35 call sites.) Memory additionally sorts `list_catalogs`/`list_tenants` while the SQL backends don't — a direct parity divergence. +- **Fix:** add `ORDER BY name` (or `id`) to every paginated query; sort memory-backend snapshots before `skip/take`. Cheap, mechanical, and testable with a "two pages cover the set exactly once" parity test. + +**B28. Search behavior diverges across all four backends.** +- LIKE/ILIKE wildcards unescaped: `postgres/main.rs:1331,1401,1438,1459`, `sqlite/business_metadata.rs:87,166,200,231` (`format!("%{}%", query)`) — a query containing `%`/`_` is a wildcard; Mongo escapes correctly (`regex::escape`), memory uses literal `contains`. Four backends, four answers. +- Tag-filter semantics: memory & SQLite = ANY-match; Postgres (`@>`) & Mongo (`$all`) = ALL-match. Empty tag list returns zero results on memory, everything on the others. +- **Fix:** escape `%`, `_`, `\` + `ESCAPE '\'` in the SQL backends; pick one tag semantic, document it on the trait (`lib.rs:490-512`), and align. + +**B29. Memory audit log: ascending order and unbounded when no filter is given.** +- `pangolin/pangolin_store/src/memory/audit.rs:14,22-23,80-85`: insertion order (oldest first) vs `ORDER BY timestamp DESC` elsewhere, and the limit/offset logic is nested inside `if let Some(filter)`, so a filterless list returns every event while SQL caps at 100. +- **Fix:** sort descending by timestamp; hoist pagination defaults out of the filter branch. + +**B30. Memory `delete_tenant` has no cascade.** +- `pangolin/pangolin_store/src/memory/tenants.rs:56-63` (`// TODO`): warehouses, catalogs, assets, and cached credentials survive tenant deletion on the memory backend. The retain-based cascade pattern already exists at `memory/catalogs.rs:107-124`. + +### High — Management UI (`pangolin_ui`) + +**B31. The API base URL env var is never defined anywhere — deployed UIs call `http://localhost:8080`.** +- `pangolin_ui/src/lib/api/client.ts:5` reads `env.PUBLIC_API_URL`; nothing sets it — `.env.example:1` declares `VITE_API_URL`, `docker-compose.yml:60` passes `VITE_API_URL`, `vitest.config.ts` sets `VITE_API_URL`. The fallback always wins, so every deployed build calls the **end user's** localhost. The login page compounds it by reading a *third* variant, `import.meta.env.VITE_API_URL || 'http://127.0.0.1:8080'` (`src/routes/login/+page.svelte:216-217`). +- **Fix:** standardize on `PUBLIC_API_URL` everywhere (SvelteKit dynamic public env requires the `PUBLIC_` prefix), default to `''` (same-origin, reverse-proxy friendly), and fail loudly in prod if unset. + +**B32. ~12 raw `fetch('/api/v1/...')` calls only work under the dev proxy — 404 in production, and omit the tenant header.** +- Sites: `src/routes/permissions/+page.svelte:34,41,68,91`; `src/routes/commits/+page.svelte:14`; `src/routes/access-requests/+page.svelte:21,37`; `src/routes/search/+page.svelte:29`; `src/routes/assets/[id]/+page.svelte:31,65`; `src/lib/components/explorer/TableDetail.svelte:57,81`; `src/lib/stores/auth.ts:43`. With `adapter-node`, relative paths hit the SvelteKit server (no `/api/v1` routes → 404); they also skip `X-Pangolin-Tenant`, so root users resolve against the wrong tenant. +- **Fix:** route all of them through `apiClient`. + +**B33. UI calls endpoints that don't exist on the server.** +- `DELETE /api/v1/branches/{catalog}/{name}` (`src/lib/api/branches.ts:64-67`) — the router (`pangolin/pangolin_api/src/lib.rs:255`) registers only GET on `/api/v1/branches/:name`; **branch deletion from the UI always 404s.** +- `GET /api/v1/users/{id}/permissions` (`src/lib/api/permissions.ts:107-111`, duplicated raw at `src/routes/permissions/+page.svelte:41`) — no such route; the permissions page and `EditPermissionsDialog.svelte:53` show an empty list forever. +- `GET /api/v1/oauth/providers` and `/api/v1/oauth/{provider}` (`src/lib/api/auth.ts:105-115`) — router has `/oauth/authorize/:provider` etc.; both 404 (currently dead code, which is why the login page hardcodes 4 provider buttons unconditionally). +- **Fix:** add the missing server routes (branch DELETE; permissions-by-user filter or `GET /api/v1/permissions?user_id=`) and wire the UI to real paths; render OAuth buttons from a real capability endpoint. + +**B34. No 401 handling anywhere; logout is client-side only.** +- `client.ts:51-60` returns errors generically; zero `401` handling in `src/` — an expired JWT leaves the user in a permanently broken "authenticated" session (`auth_token` never cleared, no redirect). Logout (`src/lib/stores/auth.ts:236-248`) only clears localStorage; `authApi.logout()`/token revocation endpoints are never called, so the JWT stays valid server-side. +- **Fix:** on 401 in `ApiClient.request`, `authStore.logout()` + `goto('/login')`; call the server revocation endpoint on logout. + +**B35. `/search` sends repeated `tags` params that the axum handler cannot deserialize.** +- `src/routes/search/+page.svelte:22-31` appends `tags` repeatedly; the server uses `Query` with `tags: Option>` (`pangolin_api/src/business_metadata_handlers.rs:189-192,205-208`) — `serde_urlencoded` can't parse repeated keys into `Vec` → HTTP 400. **Tag-filtered search is broken end-to-end** (also see B28 for the store-side divergence). +- **Fix:** comma-join tags client-side and split server-side, or switch the handler to `axum_extra::extract::Query`. + +**B36. Catalogs page: create button never renders; pagination refetches the same unpaginated list.** +- `src/lib/components/ui/DataTable.svelte:72-84`: the only `` outlet is inside `{#if searchable && !serverSide}` — the catalogs page passes `searchable={false} serverSide={true}` (`src/routes/catalogs/+page.svelte:130-131`), so the "New Catalog" control (lines 137-145) never renders and there is **no way to create a catalog from the catalogs page**. +- Same page never sends `limit`/`offset` (`+page.svelte:44-53` calls `catalogsApi.list()` bare), never sets `hasNextPage` — Next is permanently disabled. (Playwright's `test-results/.last-run.json` records 4 failed `pagination_verify` tests consistent with this.) +- Bonus: HTML comments in attribute position (`+page.svelte:127,130-135`) are parsed by Svelte as boolean props (`` etc. spread onto the component). +- **Fix:** move the actions slot outside the guard (render when `searchable || $$slots.actions`); pass `pageSize`/`offset` and set `hasNextPage = items.length === pageSize`; move comments out of attribute lists. + +**B37. Root layout references `tenantStore` without importing it; tenant switcher is dead.** +- `src/routes/+layout.svelte:88-103` uses `tenantStore.selectTenant/clearTenant` with no import (latent `ReferenceError`), the handler isn't wired to any element, and the tenant loader is commented out (lines 54-65) — root users cannot switch tenants, so `X-Pangolin-Tenant` is never set. `TenantSelector.svelte` exists and is unused. +- **Fix:** import the store, re-enable the selector. + +### Medium — Python SDK (`pypangolin`) + +**B38. `__version__ = "0.1.0"` while the package publishes 0.6.0.** +- `pypangolin/src/pypangolin/__init__.py:23` vs `pypangolin/pyproject.toml:7`. The one-version-everywhere property 0.6.0 introduced (workspace `Cargo.toml` comment) is already broken in the SDK. +- **Fix:** single-source it — `importlib.metadata.version("pypangolin")` or have CI assert equality. + +**B39. No request timeouts anywhere in the SDK — calls can hang forever.** +- `pypangolin/src/pypangolin/client.py:115`: `requests.request(method, url, headers=headers, **kwargs)` with no `timeout=`; `grep -rn timeout` over `src/` finds none. A hung server blocks the caller indefinitely (and the CLI built on it). +- **Fix:** add a configurable `timeout=(connect, read)` default (e.g. `(5, 30)`) on `PangolinClient` and pass it through; also use a `requests.Session` for connection reuse and `raise ... from e` to preserve tracebacks (`client.py:116-117`). + +**B40. `requires-python = ">=3.8"` is unsatisfiable with the declared dependencies.** +- `pypangolin/pyproject.toml:10` vs `pydantic>=2.0.0` and `pyiceberg>=0.5.0` (lines 19-27), both of which require ≥3.9 (current pyiceberg ≥3.9/3.10). A 3.8 install resolves to broken or ancient dep versions. +- **Fix:** bump to `>=3.9` (or the floor pyiceberg actually needs) and add classifiers to match. + +**B41. Importing `pypangolin` at all requires the full heavy dependency set; tests cannot even collect without it.** +- `pypangolin/src/pypangolin/__init__.py:2` → `catalog.py:1` eagerly imports `pyiceberg`. Verified: `pytest tests/` fails collection with `ModuleNotFoundError: No module named 'pyiceberg'` even though the tests under test (`tests/test_cli_config.py`, `tests/test_cli_commands.py`) only touch the CLI/config. There is also no pytest config making the `src/` layout importable without an editable install. +- **Fix:** make `get_iceberg_catalog` import pyiceberg lazily inside the function and move `pyiceberg` to an optional extra (`pypangolin[iceberg]`); add `[tool.pytest.ini_options] pythonpath = ["src"]` or document `pip install -e .` in a test README; add SDK tests to CI (see improvements). + +### High — client ↔ server contract drift (both Rust CLIs and the Python SDK) + +Neither CLI crate nor the SDK is tested against the real router, so dozens of commands call wrong endpoints or send wrong field names. They fail as 404/405/422 or — worse — as silent no-ops that print success, because every CLI command swallows exceptions with no non-zero exit (`pypangolin/src/pypangolin/cli/*.py`; `pangolin_cli_admin/src/main.rs:434` returns `Ok(())` after `eprintln!`). Representative confirmed cases (not exhaustive — ~35 sites total): + +**Rust admin CLI (`pangolin/pangolin_cli_admin/src/handlers/`)** +- **B_cli1. `create-catalog` silently creates a catalog with no warehouse and the wrong type.** `catalogs.rs:50-54` sends `{"name","warehouse","type":"pangea"}`; the server wants `warehouse_name`/`catalog_type` (`pangolin_handlers.rs:693-700`). Serde ignores the unknown fields, so `warehouse_name` is `None` (skips the existence check) and `catalog_type` defaults to `Local`; the required `--warehouse` flag is thrown away and the CLI prints success. **Fix:** `{"name","warehouse_name","catalog_type":"Local"}`. +- **B_cli2. All six merge commands hit non-existent routes.** `merge.rs:21,61,100/102,179,201,223` use `/api/v1/merges/...`; real routes are `/api/v1/catalogs/:catalog/merge-operations`, `/api/v1/merge-operations/:id[/conflicts|/complete|/abort]`, `/api/v1/conflicts/:id/resolve` (`lib.rs:262-282`). Every merge command 404s. +- **B_cli3. Four federated-catalog commands hit wrong paths; a fifth creates a `Local` catalog.** `federated.rs:12,28,142` should target `/api/v1/federated-catalogs/{name}/{sync,stats,test}` (`lib.rs:362-370`); `federated.rs:63-70` posts to `/api/v1/catalogs` with a `type` field the request struct lacks (→ `Local`); `:106` filters on `i["type"]=="federated"` where the response key is `catalog_type` with value `"Federated"` — `list-federated-catalogs` always prints empty. +- **B_cli4. Both token-revocation commands 404** (`tokens.rs:8,35` → `/api/v1/tokens/revoke*`; real routes are `/api/v1/auth/revoke*`, `lib.rs:435-442`) — the documented logout path never works. +- **B_cli5. `delete-user` passes a username where a UUID is required** (`users.rs:8-10` vs `Path` at `user_handlers.rs:381`) → 400; **`update-warehouse --id`/`update-catalog --id` key on name not id** (`warehouses.rs:241`, `catalogs.rs:92` vs `Path` name); **`revoke-permission`/`request-access` use routes/methods that don't exist** (`governance.rs:286-291,472`); **`.unwrap()` panics on JSON in `resolve_scope`** (`governance.rs:136,164,189`). +- **B_cli6. Several flags are parsed then silently dropped:** `update-user --username` (`users.rs:99`), `list-audit-events --tenant-id` (`audit.rs:28`), `resolve-conflict --merge-id` (`main.rs:310`); `assign-role`/`revoke-user-role` are unreachable non-interactively (`main.rs:432` wildcard). Multiple list commands render always-blank columns from wrong keys (`i["role"]`, `i["storage_type"]`, `i["type"]`). +- **B_cli7. `ConfigManager::new(...).unwrap()` panics** when `$HOME`/`XDG_CONFIG_HOME` are unset (`main.rs:36`; the user CLI uses `?` correctly). No request timeout on `reqwest::Client::new()` (`pangolin_cli_common/src/client.rs:13`); config file with the auth token is written `0644` (`config.rs:58-66`). + +**Rust user CLI (`pangolin_cli_user`)** +- **B_cli8.** `merge-branch` sends `source`/`target` where the server needs `source_branch`/`target_branch` → 422 (`handlers.rs:280`); `request-access` is a no-op that reports success (`handlers.rs:384-391`); `search` is a hardcoded placeholder despite two working endpoints (`handlers.rs:80`); `get-token` sends a null tenant + ignored `description` (`handlers.rs:399`); `generate-code` prints the live JWT into copy-paste output (`handlers.rs:94,117`). + +**Python SDK (`pypangolin/src/pypangolin/`)** +- **B_sdk1.** `create_user` role default `"TenantUser"` vs kebab-case `tenant-user` → 422 (`cli/admin.py:68`); `PermissionClient.grant` emits `{"type","id"}` but `PermissionScope` uses `catalog_id`/`namespace`/`asset_id`/`tag_name` → 422 (`governance.py:37-49`); `models.PermissionScope` uses kebab-case field aliases serde never emits, so scopes parse as empty (`models.py:78-83`); `Role.permissions` typed `List[Permission]` where the server returns `PermissionGrant` → `ValidationError` on any role with grants (`models.py:92`). +- **B_sdk2.** `BusinessMetadataClient.delete(asset_id, key)` deletes *all* metadata (server ignores `key`, `governance.py:136`); `request_access` sends `motivation` where the server reads `reason` (dropped, `governance.py:140`); `FederatedCatalogClient.create` drops `uri`/`warehouse`/`credential` (`federated.py:11`); `TokenClient.generate` sends `name`/`user_id`/`expires_in_days` — all ignored, token silently 24h (`admin.py:79`); `BranchClient.rebase` is missing the required `name`, ignores `base_branch`, and raises `TypeError` on the empty success body (`git.py:60-68`). +- **B_sdk3.** Broken CLI commands that crash on every call (masked by blanket `except`): `admin grant-permission` (`TypeError`, `cli/admin.py:209`), `user merge-branch` (`TypeError`, `cli/user.py:191`), `admin list-warehouses` (`AttributeError: no 'id'`, `cli/admin.py:150`), `user search` (`AttributeError: no 'score'`, `cli/user.py:80`). +- **B_sdk4. Secret handling:** connection-asset encryption key is stored inline next to its ciphertext by default (`assets/connections/base.py:68,101`); CLI writes the JWT to a `0644` `~/.pangolin/profiles.yaml` (`cli/config.py:33`); `generate-code`/`get-token` echo the raw token (`cli/user.py:94,252`). +- **B_sdk5. No console-script entry point** (`pyproject.toml` has no `[project.scripts]` despite docs calling the CLI "installed automatically"); no request timeout / `Session` reuse (`client.py:115`, `auth.py:14`); no `py.typed`; leftover `print("DEBUG CLIENT: …")` in `git.py:19`. + +Because these fail through blanket exception handlers with a zero exit code, scripts and CI cannot detect them — which is exactly why they have survived. **Fix (all):** a `wiremock`/`responses` contract-test per client method asserting path + payload against the router, `#[serde(deny_unknown_fields)]` on every server request struct so wrong field names 422 loudly, and generating the CLI/SDK request types from the existing `openapi::ApiDoc` instead of hand-writing them three times. Make every CLI command exit non-zero on failure. + +### Medium — API layer + +**B42. Management-API pagination is applied before permission filtering.** +- `pangolin/pangolin_api/src/pangolin_handlers.rs:745-786` (`list_catalogs`): the store paginates (`limit`/`offset`), then `authz_utils::filter_catalogs` removes unauthorized rows. A `TenantUser` gets variable-size pages, including **empty pages while more authorized data exists** — clients that stop on an empty page silently miss data. The same fetch-then-filter pattern applies to sibling list handlers. +- **Fix:** either filter in the store query (pass the permitted set down) or paginate after filtering; return an explicit `next_offset`/`next_page_token` so emptiness is unambiguous. + +### Low — hygiene & docs + +**B43. `docs/environment-variables.md` documents variables that don't exist and misses ~20 that do.** +- It lists `PANGOLIN_HOST`, `PANGOLIN_PORT`, `PANGOLIN_STORE_TYPE` — none are read by the server (the real set includes `PANGOLIN_BIND_ADDRESS`, `PANGOLIN_STORAGE_TYPE`, `PANGOLIN_MAX_BODY_BYTES`, `PANGOLIN_REQUEST_TIMEOUT_SECS`, `PANGOLIN_CORS_ALLOWED_ORIGINS`, `PANGOLIN_WAREHOUSE_CACHE_TTL_SECS`, `PANGOLIN_SESSION_TTL_SECS`, `PANGOLIN_METRICS_ENABLED`, all OAuth vars, etc. — verified against `grep "PANGOLIN_" pangolin_api/src`). `docs/getting-started/env_vars.md` is closer but also incomplete. +- **Fix:** regenerate the reference from `pangolin_api/src/config.rs` (single source of truth); delete or redirect the stale file. + +**B44. A live PyPI API token sits in plaintext in the repo-root `.env`.** +- `/home/alexmerced/development/personal/Personal/library/2026/pangolin/.env` contains `PYPI_token=pypi-AgEI...`. Verified **not** tracked by git and never in history — but it is one `.gitignore` edit (or one `git add -f`) away from leaking, and any local tool with repo read access can exfiltrate it. +- **Fix:** rotate the token now, remove it from `.env`, and keep publish credentials in a keyring / CI secret (`pypangolin/PUBLISHING.md` already describes the `secrets.PYPI_API_TOKEN` flow). + +**B45. Debug debris is committed to git.** +- Tracked: `pangolin/.test_output.txt`, `pangolin/logs/api_log.txt`, `pangolin_ui/check_catalogs_list.txt`, `check_fed_cat.txt`, `check_final.txt`, `check_service_users*.txt` (~260 KB), plus untracked-but-present `pangolin/logs/*.log`, `video/render.log`, and two stale monoliths `pangolin_store/src/memory.rs.bak` / `mongo.rs.bak` (~4k lines of divergent query copies — a grep trap). +- **Fix:** `git rm --cached` the tracked ones (ignore patterns already exist; they were added before the ignore), delete the `.bak` files, add `logs/`, `test-results/`, `playwright-report/` to `pangolin_ui/.gitignore`. + +**B46. UI package/test plumbing defects.** +- `package-lock.json:3` still says `0.1.0` (root `package.json` is 0.6.0); `@vitest/coverage-v8` missing so `npm run test:coverage` fails; no `test:e2e` script despite Playwright specs; `playwright.config.ts:11` targets port 5175 while `vite dev` serves 5173 and `webServer` is commented out; 18 unused runtime deps (all `@smui/*`, `marked`, `material-icons`) bloating the Docker image; `client.test.ts:17-20` mocks lack `.text()` so all 5 "passing" tests actually exercise the error path; `src/tests/CatalogsList.test.ts:44-47` asserts on a button that no longer exists. +- **Fix:** regenerate the lockfile, add the coverage dep and `test:e2e` script, align Playwright port + enable `webServer`, prune deps, fix the fetch mocks. + +--- + +## Recommended Improvements + +Ordered by leverage. + +0. **Add a permission-matrix (authz) test and a client↔server contract test — the two highest-leverage additions.** Every bug in the "API authorization bypasses" cluster (B0a-B0m) and the "contract drift" cluster (B_cli*/B_sdk*) is invisible to the current CI: it compiles, it's formatted, it's lint-clean, and the unit tests pass — because nothing asserts *who is allowed to call what* or *whether a client's request matches the router*. Add (a) a table-driven test that, for each mounted route, drives it as `Root`/`TenantAdmin`/`TenantUser`/wrong-tenant/service-user and asserts the expected 200/403 — this catches every missing `check_permission`; and (b) a `wiremock`/`responses` contract suite for both CLIs and the SDK that asserts each method's path + payload against the real handler. Pair with `#[serde(deny_unknown_fields)]` on every server request struct so wrong field names fail loudly instead of defaulting. + +1. **Build a cross-backend parity test harness and make it the gate.** Nearly half the bugs above (B1-B7, B17-B30) are one backend silently diverging from the others. A single test suite that runs every `CatalogStore` method against all four backends and asserts identical observable behavior (including sort order, pagination determinism via the "two pages = whole set" property, serde round-trips of *every* enum variant, and tenant isolation on every read) would have caught all of them and will keep them fixed. Wire it into the existing `ci.yml` services matrix. `docs/operations/backend-parity.md` can then be generated from the suite instead of maintained by hand. +2. **Add pypangolin and pangolin_ui jobs to CI.** `.github/workflows/ci.yml` covers only Rust, Helm, and Docker. Add: `pytest` (after B41 makes collection possible), `ruff` + `mypy` for the SDK; `npm run check`, `vitest`, and Playwright (after B46) for the UI. The UI/SDK are exactly where 0.6.0's "no CI → silent rot" lesson is currently repeating. +3. **Fail-fast config validation for deployment artifacts.** A CI step that runs `docker compose config` on every compose file and boots the API container with the compose-provided env (catching B8/B9/B10-class drift), plus a script that greps compose/Helm/docs for `PANGOLIN_*` names and diffs them against `config.rs`. +4. **Unify the error envelope.** Management API emits flat `{"error": ""}` while Iceberg handlers emit the spec envelope; the UI already guesses (`errorData.error || errorData.message`, `client.ts:54`) and renders `[object Object]` for structured errors. Standardize on one management envelope (message + code + request_id), and finish migrating handlers off bare `(StatusCode, &str)` tuples. +5. **Reduce panic surface.** 92 non-test `unwrap()` in `pangolin_api`, 38 in `pangolin_store`, including 14 `unwrap()`s on database-supplied BSON in the Mongo backend (`mongo/branches.rs:58-105`, `mongo/assets.rs:82-190`, `mongo/business_metadata.rs:104-218`, etc.) where a single legacy document panics the request. Convert to typed errors; then drive the clippy budget from 36 to 0 and flip CI to `-D warnings`. +6. **Token handling:** store only hashes of session tokens at rest (`sqlite/tokens.rs:11-19`, `mongo/tokens.rs:66-75` currently persist and even return raw tokens via `list_active_tokens`) — the `service_users.api_key_hash` pattern already exists. On the UI side, move toward an `httpOnly` cookie session (or at minimum short TTL + revoke-on-logout) instead of long-lived JWTs in `localStorage`. +7. **SDK ergonomics:** `requests.Session` with retry/backoff (429/5xx), `raise ... from e`, Pydantic `ConfigDict` migration (deprecation warnings at `models.py:132,196`), and typed pagination helpers that iterate all pages. +8. **Docs generation over hand maintenance:** env-var reference from `config.rs` (B43), Iceberg endpoint coverage table from the router, and a single CHANGELOG-driven version bump script that touches `Cargo.toml`, `pyproject.toml`, `__init__.py`, `package.json` + lockfile, and the Helm chart in one commit (B38/B46 show the manual process already drifting one day after 0.6.0). +9. **Known-limitation burndown (README already admits these; they are the right next hardening targets):** per-IP/per-account rate limiting on `/api/v1/users/login` and the token endpoints (e.g. `tower_governor`); encrypt warehouse cloud credentials at rest (envelope encryption, KMS-or-env master key); make branch-create-by-copy transactional on Postgres. +10. **UI cleanup pass:** remove dead code (`src/lib/stores.ts`, `app.scss`, the unauthenticated file-read route `src/routes/api/docs/[...path]/+server.ts` — broken in Docker, no callers, blacklist-based traversal guard), strip ~25 `console.log`s (one logs a token prefix, `stores/auth.ts:76`), replace native `alert()`/`confirm()` with the existing `ConfirmDialog`, and delete the "unresolved deliberation" comments that describe the very bugs above (`routes/login/+page.svelte:204-215`, `routes/permissions/+page.svelte:6-12`). + +--- + +## Recommended New Features + +1. **Complete the Iceberg REST surface** (the README's own "not implemented" list): `loadNamespaceMetadata` (GET namespace), `namespaceExists` (HEAD), `registerTable`, `commitTransaction` (multi-table atomic commits), and the rest of the view API (list/drop/replace/exists/rename). `registerTable` + full views are the cheapest wins; `commitTransaction` builds directly on the now-solid `commit.rs` requirement machinery and would be a genuine differentiator at this maturity level. +2. **Table maintenance as a first-class service:** scheduled snapshot expiration, orphan-file detection/cleanup, and manifest compaction driven from `pangolin_core/src/maintenance.rs`, surfaced in the UI and CLI. Pairs naturally with fixing `metadata-log` (B13) since expiration needs the metadata history. +3. **Backup/restore + disaster recovery tooling:** a `pangolin-admin backup`/`restore` command per backend (pg_dump-wrapping for Postgres, file copy for SQLite, mongodump for Mongo) with a documented, *tested* RPO/RTO — the README currently ships "undocumented and untested" for this row, and several bugs above (B3, B21) make backups the only recovery path. +4. **Shared-state option for multi-replica deployments:** a Redis (or Postgres-advisory-lock) backing for the warehouse cache, OAuth nonce store, and token-cleanup job coordination — converting the README's "HA unproven" row into a supported topology. The cache already documents its node-locality (`cached_store.rs:30,45`); this is the designed-for next step. +5. **Full OIDC:** PKCE, `id_token` validation via JWKS, provider discovery, and an `email_verified` check — closing the documented OAuth gaps in `docs/operations/oidc.md` and letting the UI's provider buttons render from a real `GET /api/v1/oauth/providers` capability endpoint (B33). +6. **Webhooks / event stream on catalog changes:** the audit pipeline already classifies 40+ actions across 19 resource types; emitting them to configurable HTTP/queue sinks enables cache invalidation for downstream engines, data-product notifications, and CDC-style integration — a feature none of the small OSS catalogs do well. +7. **Real search:** replace `LIKE '%q%'` with Postgres `pg_trgm`/`tsvector` (and equivalent per-backend strategies) with ranked results, and expose facets (asset type, tags, namespace) — the business-catalog feature set already stores the metadata; the query layer is the missing piece (and B28 shows the current one is inconsistent anyway). +8. **Load-test harness + published capacity numbers:** a `k6`/`goose` scenario pack (login, listing, commit contention, credential vending) run in CI-nightly, publishing p50/p99 and a measured capacity model — directly addresses the README's "no published performance figures" and would catch regressions like the pre-0.6.0 bcrypt scan. +9. **Soft-delete / recycle bin for catalogs and branches:** given that cascading deletes are the most dangerous operations in the system (B3, B21, and the documented non-atomic branch copy), a two-phase delete (mark, then purge after N days) is cheap insurance and a strong operator-trust feature. +10. **`pypangolin` async client (`httpx.AsyncClient`)** sharing the same models, plus a documented PyIceberg-compat test matrix in CI (the emulator compose file already exists at `docker-compose.emulators.yml` — wire it to a nightly job). + +--- + +*Method note: findings above were verified directly against the working tree (file:line citations checked on 2026-08-10), with `cargo test --workspace`, `cargo fmt --check`, `cargo clippy` executed locally, plus targeted deep-dives across `pangolin_store` (all four backends), `pangolin_api` (Iceberg + management handlers), `pangolin_ui`, and `pypangolin`.* From bdcfdb0a422c110f13e84c9b6bb06cdbdd9bbab4 Mon Sep 17 00:00:00 2001 From: Alex Merced Date: Mon, 10 Aug 2026 10:28:15 -0400 Subject: [PATCH 02/23] fix(store): align the four backends and add a cross-backend parity suite Addresses the storage-layer cluster of roadmap_aug10.md (B1-B7, B17-B30) plus improvement #1, the parity harness that makes the whole class of finding testable rather than reviewable. Tenant isolation and data integrity B1 Mongo's get_audit_event discarded the tenant_id parameter, so any tenant holding an audit UUID could read another tenant's record. B2 Mongo revocation was a silent no-op: the write went through serde as a string under `id`, the check queried a BSON UUID under `token_id`. Nothing could ever match, so revoked JWTs - including after logout - stayed valid. Cleanup had the mirror type mismatch and deleted nothing, so the collection grew without bound. B3 SQLite delete_branch referenced a column that does not exist and was non-transactional, so the branch was committed away and its assets orphaned while the caller saw an error. B4 Postgres decoded a TEXT[] as String; `Row::get` panics on decode failure, so any search with a hit panicked the request. SQLite's sibling silently returned raw JSON as the namespace. B5 Mongo dropped the compare-and-swap entirely, so two concurrent Iceberg commits both "succeeded" and one snapshot was lost. B6 The memory by-id asset index was keyed on catalog name alone, so tenant A deleting `sales` broke get_asset_by_id for tenant B's. B7 All three persistent backends stored the Debug spelling of AssetType and parsed only two variants, defaulting the other 15 to IcebergTable - a DeltaTable round-tripped as an Iceberg table. B26 Memory skipped the CAS whenever the expectation was None, which is the create-path assertion, not "no check". B30 Memory delete_tenant had no cascade: warehouses (with credentials), catalogs, assets and tokens all outlived the tenant. Parity and determinism B17 One ns_key helper on memory; create/get keyed on "." while delete/update keyed on 0x1F, so nested namespaces were undeletable. B18 SQLite's foreign_keys pragma is per connection but was set on one arbitrary pooled connection, making ON DELETE CASCADE fire nondeterministically. Now set through the connect options. B19 SQLite get_metadata_location ignored the branch and returned an arbitrary row, so a dev read could return main's pointer. B20 SQLite left the metadata_location column stale, freezing Asset.location at creation time after every commit. B21 SQLite delete_catalog cascaded before checking existence, so deleting a nonexistent catalog destroyed matching children first. B22 SQLite persisted Debug action names and parsed snake_case, then swallowed the mismatch, misattributing nearly every audited action. B23 Mongo audit filters compared Debug names to snake_case documents and always matched zero rows; listings were unsorted and unbounded. B24 Postgres list_catalogs omitted catalog_type/federated_config from the SELECT and hardcoded Local, so federated catalogs looked local. B25 Memory merge_branch reused asset ids (repointing the by-id index at the copy) and never advanced the target head. B27 ORDER BY on every paginated query; memory sorts before slicing. B28 One definition of search: LIKE metacharacters escaped, and one tag semantic (ALL-match, empty list means no filter) across all four. B29 Memory audit returned oldest-first and unbounded without a filter. Two defects the new parity suite found on its first run * SqliteStore had no inherent revoke_token/is_token_revoked, so the trait delegations in sqlite/main.rs called themselves. Revoking a token on SQLite recursed until the stack was exhausted and aborted the process. * The SQLite audit_logs table still declared the original (actor, resource, details) shape while the code inserted the full AuditLogEntry, so every audit write failed with "no such column" and the backend kept no audit trail at all. Schema version bumped to 2. New CatalogStore methods delete_file and replace_namespace_properties are implemented across all four backends. cargo test --workspace: 57 targets green. Clippy budget lowered 36 -> 34. Co-Authored-By: Claude Opus 5 --- pangolin/clippy-warning-budget.txt | 2 +- pangolin/pangolin_core/src/audit.rs | 49 ++ pangolin/pangolin_core/src/model.rs | 67 ++ pangolin/pangolin_store/sql/sqlite_schema.sql | 23 +- pangolin/pangolin_store/src/lib.rs | 1 + .../src/memory/access_requests.rs | 8 +- pangolin/pangolin_store/src/memory/assets.rs | 40 +- pangolin/pangolin_store/src/memory/audit.rs | 30 +- .../pangolin_store/src/memory/branches.rs | 29 +- .../pangolin_store/src/memory/catalogs.rs | 14 +- pangolin/pangolin_store/src/memory/io.rs | 37 +- pangolin/pangolin_store/src/memory/main.rs | 43 +- .../pangolin_store/src/memory/namespaces.rs | 9 +- .../pangolin_store/src/memory/permissions.rs | 8 +- pangolin/pangolin_store/src/memory/roles.rs | 8 +- .../src/memory/service_users.rs | 8 +- pangolin/pangolin_store/src/memory/tags.rs | 8 +- pangolin/pangolin_store/src/memory/tenants.rs | 39 +- pangolin/pangolin_store/src/memory/tokens.rs | 10 +- pangolin/pangolin_store/src/memory/users.rs | 8 +- .../pangolin_store/src/memory/warehouses.rs | 9 +- pangolin/pangolin_store/src/mongo/assets.rs | 23 +- pangolin/pangolin_store/src/mongo/audit.rs | 55 +- pangolin/pangolin_store/src/mongo/main.rs | 39 +- pangolin/pangolin_store/src/mongo/mod.rs | 6 +- pangolin/pangolin_store/src/mongo/tokens.rs | 41 +- .../pangolin_store/src/postgres/assets.rs | 26 +- .../pangolin_store/src/postgres/branches.rs | 2 +- .../pangolin_store/src/postgres/catalogs.rs | 23 +- pangolin/pangolin_store/src/postgres/main.rs | 33 +- .../pangolin_store/src/postgres/namespaces.rs | 2 +- .../src/postgres/permissions.rs | 2 +- pangolin/pangolin_store/src/postgres/roles.rs | 2 +- pangolin/pangolin_store/src/postgres/tags.rs | 2 +- .../pangolin_store/src/postgres/tenants.rs | 12 +- pangolin/pangolin_store/src/postgres/users.rs | 4 +- .../pangolin_store/src/postgres/warehouses.rs | 2 +- pangolin/pangolin_store/src/search.rs | 94 +++ .../src/sqlite/access_requests.rs | 15 +- pangolin/pangolin_store/src/sqlite/assets.rs | 49 +- .../pangolin_store/src/sqlite/audit_logs.rs | 44 +- .../pangolin_store/src/sqlite/branches.rs | 32 +- .../src/sqlite/business_metadata.rs | 43 +- .../pangolin_store/src/sqlite/catalogs.rs | 70 +- pangolin/pangolin_store/src/sqlite/main.rs | 41 +- .../src/sqlite/merge_operations.rs | 4 +- .../pangolin_store/src/sqlite/namespaces.rs | 2 +- .../pangolin_store/src/sqlite/permissions.rs | 2 +- pangolin/pangolin_store/src/sqlite/roles.rs | 2 +- .../src/sqlite/service_users.rs | 2 +- pangolin/pangolin_store/src/sqlite/tags.rs | 2 +- pangolin/pangolin_store/src/sqlite/tenants.rs | 11 +- pangolin/pangolin_store/src/sqlite/tokens.rs | 54 +- pangolin/pangolin_store/src/sqlite/users.rs | 4 +- .../pangolin_store/src/sqlite/warehouses.rs | 2 +- pangolin/pangolin_store/src/tests/mod.rs | 2 + pangolin/pangolin_store/src/tests/parity.rs | 764 ++++++++++++++++++ .../pangolin_store/tests/mongo_audit_tests.rs | 12 +- .../pangolin_store/tests/store_integration.rs | 6 +- 59 files changed, 1649 insertions(+), 332 deletions(-) create mode 100644 pangolin/pangolin_store/src/search.rs create mode 100644 pangolin/pangolin_store/src/tests/parity.rs diff --git a/pangolin/clippy-warning-budget.txt b/pangolin/clippy-warning-budget.txt index 7facc89..a787364 100644 --- a/pangolin/clippy-warning-budget.txt +++ b/pangolin/clippy-warning-budget.txt @@ -1 +1 @@ -36 +34 diff --git a/pangolin/pangolin_core/src/audit.rs b/pangolin/pangolin_core/src/audit.rs index 2091388..1f3ce09 100644 --- a/pangolin/pangolin_core/src/audit.rs +++ b/pangolin/pangolin_core/src/audit.rs @@ -206,6 +206,55 @@ pub enum AuditResult { Failure, } +/// Serialize an audit enum to the spelling that goes into storage. +/// +/// B22: the SQLite backend persisted `format!("{:?}", action)` - the Debug +/// spelling, `"CreateBranch"` - and then read it back by lowercasing to +/// `"createbranch"` and deserializing against serde's snake_case +/// (`"create_branch"`). Nothing matched, and the result was +/// `.unwrap_or(AuditAction::CreateCatalog)`, so nearly every multi-word action +/// in the SQLite audit trail was recorded as `CreateCatalog`. The audit log is +/// the one artefact that has to be right after an incident. +/// +/// Both directions now go through serde, so the write and read spellings cannot +/// drift apart again. +pub fn audit_enum_to_stored(value: &T) -> String { + serde_json::to_value(value) + .ok() + .and_then(|v| v.as_str().map(str::to_owned)) + .unwrap_or_else(|| format!("{:?}", value)) +} + +/// Parse an audit enum from its stored spelling. +/// +/// Accepts the canonical serde name and, for rows written before B22 was fixed, +/// the legacy Debug name. An unrecognised value is an error rather than a silent +/// substitution. +pub fn audit_enum_from_stored(stored: &str) -> Result +where + T: serde::de::DeserializeOwned, +{ + // Canonical snake_case. + if let Ok(v) = serde_json::from_value::(serde_json::Value::String(stored.to_string())) { + return Ok(v); + } + + // Legacy Debug spelling: "CreateBranch" -> "create_branch". + let mut snake = String::with_capacity(stored.len() + 4); + for (i, ch) in stored.chars().enumerate() { + if ch.is_ascii_uppercase() { + if i != 0 { + snake.push('_'); + } + snake.push(ch.to_ascii_lowercase()); + } else { + snake.push(ch); + } + } + serde_json::from_value::(serde_json::Value::String(snake)) + .map_err(|_| format!("unknown audit enum value {stored:?}")) +} + /// Filter for querying audit logs #[derive(Debug, Clone, Deserialize)] #[cfg_attr(feature = "utoipa", derive(ToSchema))] diff --git a/pangolin/pangolin_core/src/model.rs b/pangolin/pangolin_core/src/model.rs index 0b94f89..ca18774 100644 --- a/pangolin/pangolin_core/src/model.rs +++ b/pangolin/pangolin_core/src/model.rs @@ -111,6 +111,73 @@ pub enum AssetType { Other, } +impl AssetType { + /// The canonical persisted spelling, e.g. `"DELTA_TABLE"`. + /// + /// B7: all three persistent backends wrote `format!("{:?}", asset.kind)` - + /// the *Debug* spelling - and read it back through a match that recognised + /// only `IcebergTable` and `View`, defaulting everything else to + /// `IcebergTable`. A `DeltaTable`, `MlModel`, `Lance` or any of the other + /// 13 variants round-tripped as an Iceberg table, silently defeating the + /// headline "tracks any lakehouse asset type" feature. Going through serde + /// means the enum's own rename policy is the single source of truth, and + /// adding a variant cannot reintroduce the drift. + pub fn as_stored_str(&self) -> String { + serde_json::to_value(self) + .ok() + .and_then(|v| v.as_str().map(str::to_owned)) + // The enum is a plain unit-variant enum, so serialization cannot + // fail; the Debug spelling is a belt-and-braces fallback only. + .unwrap_or_else(|| format!("{:?}", self)) + } + + /// Parse a persisted asset type. + /// + /// Accepts the canonical serde spelling *and* the legacy Debug spelling, so + /// rows written before this fix keep loading. An unrecognised value is an + /// error rather than a silent downgrade - a wrong asset type is worse than a + /// loud failure, because it misroutes every reader downstream. + pub fn from_stored_str(value: &str) -> Result { + if let Ok(parsed) = + serde_json::from_value::(serde_json::Value::String(value.to_string())) + { + return Ok(parsed); + } + + // Legacy Debug spellings, e.g. "IcebergTable". + for candidate in Self::all() { + if format!("{:?}", candidate) == value { + return Ok(candidate); + } + } + + Err(format!("unknown asset type {value:?}")) + } + + /// Every variant, for round-trip tests and legacy parsing. + pub fn all() -> Vec { + vec![ + Self::IcebergTable, + Self::DeltaTable, + Self::HudiTable, + Self::ParquetTable, + Self::CsvTable, + Self::JsonTable, + Self::View, + Self::MlModel, + Self::ApachePaimon, + Self::Vortex, + Self::Lance, + Self::Nimble, + Self::Directory, + Self::VideoFile, + Self::ImageFile, + Self::DbConnString, + Self::Other, + ] + } +} + #[derive(Debug, Clone, Serialize, Deserialize)] pub struct Asset { pub id: Uuid, diff --git a/pangolin/pangolin_store/sql/sqlite_schema.sql b/pangolin/pangolin_store/sql/sqlite_schema.sql index f3d7ab5..46bf7fb 100644 --- a/pangolin/pangolin_store/sql/sqlite_schema.sql +++ b/pangolin/pangolin_store/sql/sqlite_schema.sql @@ -113,14 +113,29 @@ CREATE TABLE IF NOT EXISTS commits ( CREATE INDEX IF NOT EXISTS idx_commits_tenant ON commits(tenant_id); -- Audit Logs +-- The columns here had drifted out of step with the code that writes them. +-- The table declared the original (actor, resource, details) shape while +-- `sqlite/audit_logs.rs` inserted the full AuditLogEntry - user_id, username, +-- resource_type, resource_id, resource_name, ip_address, user_agent, result, +-- error_message, metadata. Every `log_audit_event` on SQLite therefore failed +-- at runtime with "table audit_logs has no column named user_id", so the +-- backend recorded *no* audit trail at all. Found by the cross-backend parity +-- suite, which is the first thing to exercise audit logging on SQLite. CREATE TABLE IF NOT EXISTS audit_logs ( id TEXT PRIMARY KEY, tenant_id TEXT NOT NULL, - timestamp INTEGER NOT NULL, - actor TEXT NOT NULL, + user_id TEXT, + username TEXT NOT NULL, action TEXT NOT NULL, - resource TEXT NOT NULL, - details TEXT, -- JSON + resource_type TEXT NOT NULL, + resource_id TEXT, + resource_name TEXT NOT NULL, + timestamp INTEGER NOT NULL, + ip_address TEXT, + user_agent TEXT, + result TEXT NOT NULL, + error_message TEXT, + metadata TEXT, -- JSON FOREIGN KEY (tenant_id) REFERENCES tenants(id) ON DELETE CASCADE ); CREATE INDEX IF NOT EXISTS idx_audit_logs_tenant_ts ON audit_logs(tenant_id, timestamp DESC); diff --git a/pangolin/pangolin_store/src/lib.rs b/pangolin/pangolin_store/src/lib.rs index aa6d1db..05427c2 100644 --- a/pangolin/pangolin_store/src/lib.rs +++ b/pangolin/pangolin_store/src/lib.rs @@ -29,6 +29,7 @@ pub use sqlite::SqliteStore; pub mod metadata_cache; pub mod object_store_cache; pub mod object_store_factory; +pub mod search; pub use metadata_cache::MetadataCache; pub use object_store_cache::ObjectStoreCache; pub use signer::SignerImpl; diff --git a/pangolin/pangolin_store/src/memory/access_requests.rs b/pangolin/pangolin_store/src/memory/access_requests.rs index 788180f..c7b3514 100644 --- a/pangolin/pangolin_store/src/memory/access_requests.rs +++ b/pangolin/pangolin_store/src/memory/access_requests.rs @@ -44,13 +44,7 @@ impl MemoryStore { .filter(|req| req.value().tenant_id == tenant_id) .map(|req| req.value().clone()); - let requests = if let Some(p) = pagination { - iter.skip(p.offset.unwrap_or(0)) - .take(p.limit.unwrap_or(usize::MAX)) - .collect() - } else { - iter.collect() - }; + let requests = crate::memory::main::paginate_sorted(iter, pagination, |r| r.id); Ok(requests) } } diff --git a/pangolin/pangolin_store/src/memory/assets.rs b/pangolin/pangolin_store/src/memory/assets.rs index bd0b006..621a186 100644 --- a/pangolin/pangolin_store/src/memory/assets.rs +++ b/pangolin/pangolin_store/src/memory/assets.rs @@ -29,6 +29,7 @@ impl MemoryStore { self.assets_by_id.insert( asset.id, ( + tenant_id, catalog_name.to_string(), namespace.clone(), Some(branch_name.clone()), @@ -83,8 +84,12 @@ impl MemoryStore { asset_id: Uuid, ) -> Result)>> { if let Some(entry) = self.assets_by_id.get(&asset_id) { - let (catalog_name, namespace, branch, name) = entry.value().clone(); - // Verify tenant ownership (implicit via proper key lookup) purely for safety + let (owner_tenant, catalog_name, namespace, branch, name) = entry.value().clone(); + // The index is global across tenants, so ownership is checked here + // rather than relying on the composite key lookup below to miss. + if owner_tenant != tenant_id { + return Ok(None); + } let branch_name = branch.unwrap_or_else(|| "main".to_string()); let key = ( tenant_id, @@ -120,13 +125,8 @@ impl MemoryStore { }) .map(|entry| entry.value().clone()); - let assets: Vec = if let Some(p) = pagination { - iter.skip(p.offset.unwrap_or(0)) - .take(p.limit.unwrap_or(usize::MAX)) - .collect() - } else { - iter.collect() - }; + let assets: Vec = + crate::memory::main::paginate_sorted(iter, pagination, |a| a.name.clone()); Ok(assets) } pub(crate) async fn delete_asset_internal( @@ -184,6 +184,7 @@ impl MemoryStore { self.assets_by_id.insert( asset.id, ( + tenant_id, catalog_name.to_string(), dest_namespace, Some(branch_val), @@ -245,15 +246,17 @@ impl MemoryStore { false }; - let tags_match = if let Some(ref search_tags) = tags { - if let Some(ref meta) = metadata { - search_tags.iter().any(|tag| meta.tags.contains(tag)) - } else { - false - } - } else { - true - }; + // B28: this was an ANY-match, while Postgres (`@>`) and Mongo + // (`$all`) required all requested tags - and an *empty* tag list + // returned nothing here but everything on the other three. + // `crate::search::tags_match` is the single definition of the + // chosen ALL-match semantic, with an empty list meaning "no tag + // filter". + let owned_tags = metadata + .as_ref() + .map(|m| m.tags.clone()) + .unwrap_or_default(); + let tags_match = crate::search::tags_match(&owned_tags, tags.as_deref()); if (name_matches || description_matches) && tags_match { let namespace: Vec = @@ -316,6 +319,7 @@ impl MemoryStore { self.assets_by_id.insert( asset.id, ( + tenant_id, catalog_name.to_string(), ns.clone(), Some(dest_branch.to_string()), diff --git a/pangolin/pangolin_store/src/memory/audit.rs b/pangolin/pangolin_store/src/memory/audit.rs index f80d834..f26d385 100644 --- a/pangolin/pangolin_store/src/memory/audit.rs +++ b/pangolin/pangolin_store/src/memory/audit.rs @@ -2,6 +2,9 @@ use super::MemoryStore; use anyhow::Result; use uuid::Uuid; +/// Default cap on a listing, matching the SQL backends' `LIMIT 100`. +const DEFAULT_AUDIT_LIMIT: usize = 100; + impl MemoryStore { pub(crate) async fn log_audit_event_internal( &self, @@ -19,6 +22,14 @@ impl MemoryStore { tenant_id: Uuid, filter: Option, ) -> Result> { + // Read the pagination window out before the filter is consumed below. + let pagination = filter.as_ref().map(|f| { + ( + f.offset.unwrap_or(0), + f.limit.unwrap_or(DEFAULT_AUDIT_LIMIT), + ) + }); + if let Some(events) = self.audit_events.get(&tenant_id) { let mut filtered = events.clone(); @@ -76,13 +87,22 @@ impl MemoryStore { true }); + } - // Apply pagination - let offset = f.offset.unwrap_or(0); - let limit = f.limit.unwrap_or(100); + // B29: two divergences from the SQL backends, both fixed here. + // + // 1. Events were returned in *insertion* order (oldest first) while + // every SQL backend uses `ORDER BY timestamp DESC`. A caller + // asking for "the last 100 events" got the *first* 100. + // 2. Pagination lived inside the `if let Some(filter)` block, so a + // filterless listing returned the tenant's entire audit history + // while the SQL backends capped it at 100. On a busy tenant that + // is an unbounded allocation driven by an unauthenticated-shaped + // call pattern. + filtered.sort_by(|a, b| b.timestamp.cmp(&a.timestamp)); - filtered = filtered.into_iter().skip(offset).take(limit).collect(); - } + let (offset, limit) = pagination.unwrap_or((0, DEFAULT_AUDIT_LIMIT)); + let filtered = filtered.into_iter().skip(offset).take(limit).collect(); Ok(filtered) } else { diff --git a/pangolin/pangolin_store/src/memory/branches.rs b/pangolin/pangolin_store/src/memory/branches.rs index ef8cc15..4171a85 100644 --- a/pangolin/pangolin_store/src/memory/branches.rs +++ b/pangolin/pangolin_store/src/memory/branches.rs @@ -62,13 +62,8 @@ impl MemoryStore { }) .map(|entry| entry.value().clone()); - let branches: Vec = if let Some(p) = pagination { - iter.skip(p.offset.unwrap_or(0)) - .take(p.limit.unwrap_or(usize::MAX)) - .collect() - } else { - iter.collect() - }; + let branches: Vec = + crate::memory::main::paginate_sorted(iter, pagination, |b| b.name.clone()); Ok(branches) } pub(crate) async fn delete_branch_internal( @@ -101,7 +96,8 @@ impl MemoryStore { source_branch_name: String, target_branch_name: String, ) -> Result<()> { - self.get_branch_internal(tenant_id, catalog_name, source_branch_name.clone()) + let source_branch = self + .get_branch_internal(tenant_id, catalog_name, source_branch_name.clone()) .await? .ok_or_else(|| anyhow::anyhow!("Source branch '{}' not found", source_branch_name))?; @@ -126,21 +122,34 @@ impl MemoryStore { let namespace_parts: Vec = namespace_key.split('\x1F').map(|s| s.to_string()).collect(); + // B25: the copy used to keep `asset.id`. `create_asset_internal` + // writes `assets_by_id[asset.id]`, so the shared id was repointed at + // the *target*-branch copy and every subsequent `get_asset_by_id` + // for the source branch's asset resolved to the wrong branch. A + // merged copy is a distinct row and needs a distinct identity. + let mut copied = asset.clone(); + copied.id = Uuid::new_v4(); + self.create_asset_internal( tenant_id, catalog_name, Some(target_branch_name.clone()), namespace_parts.clone(), - asset.clone(), + copied.clone(), ) .await?; - let qualified = format!("{}.{}", namespace_parts.join("."), asset.name); + let qualified = format!("{}.{}", namespace_parts.join("."), copied.name); if !target_branch.assets.contains(&qualified) { target_branch.assets.push(qualified); } } + // B25: the target's head was never advanced, unlike all three other + // backends, so after a merge the target branch still pointed at its + // pre-merge commit. + target_branch.head_commit_id = source_branch.head_commit_id; + self.create_branch_internal(tenant_id, catalog_name, target_branch) .await?; diff --git a/pangolin/pangolin_store/src/memory/catalogs.rs b/pangolin/pangolin_store/src/memory/catalogs.rs index f91401c..d29163e 100644 --- a/pangolin/pangolin_store/src/memory/catalogs.rs +++ b/pangolin/pangolin_store/src/memory/catalogs.rs @@ -118,10 +118,16 @@ impl MemoryStore { // Remove Tags self.tags.retain(|k, _| !(k.0 == tenant_id && k.1 == name)); - // Clean up assets_by_id index - // This is expensive O(N) since we have to scan the whole index - // But deletion is rare. - self.assets_by_id.retain(|_, v| v.0 != name); + // Clean up assets_by_id index. + // + // B6: this filtered on the catalog *name* alone, so deleting + // tenant A's `sales` catalog also evicted tenant B's `sales` assets + // from the by-id index - `get_asset_by_id` then returned `None` for + // rows that were still perfectly present. The index value now + // carries the tenant, so the cascade matches on both. + // O(N) over the index, but catalog deletion is rare. + self.assets_by_id + .retain(|_, v| !(v.0 == tenant_id && v.1 == name)); Ok(()) } else { diff --git a/pangolin/pangolin_store/src/memory/io.rs b/pangolin/pangolin_store/src/memory/io.rs index f5b7daa..6e9bd59 100644 --- a/pangolin/pangolin_store/src/memory/io.rs +++ b/pangolin/pangolin_store/src/memory/io.rs @@ -52,21 +52,38 @@ impl MemoryStore { ); if let Some(mut asset) = self.assets.get_mut(&key) { + // Resolved exactly as `get_metadata_location_internal` resolves it - + // property first, then the asset's own location - so the CAS + // compares against the value a reader would have seen. This mirrors + // SQLite, where the fallback is the `metadata_location` column. let current_loc = asset .properties .get("metadata_location") .cloned() - .unwrap_or(asset.location.clone()); + .or_else(|| { + if asset.location.is_empty() { + None + } else { + Some(asset.location.clone()) + } + }); - // CAS Check - if let Some(expected) = expected_location { - if current_loc != expected { - return Err(anyhow::anyhow!( - "CAS failure: expected {} but found {}", - expected, - current_loc - )); - } + // CAS check. + // + // B26: this used to be `if let Some(expected) = expected_location`, + // which skipped the check entirely when the caller passed `None`. + // But `None` is not "don't check" - it is the *create-path* + // assertion "there must be no metadata location yet". Skipping it + // meant a create-table race that Postgres and SQLite correctly + // rejected silently succeeded in dev and in every memory-backed + // test, which is precisely where such a race would have been caught. + // The unconditional comparison matches the SQLite form. + if current_loc != expected_location { + return Err(anyhow::anyhow!( + "CAS failure: expected {:?} but found {:?}", + expected_location, + current_loc + )); } asset.location = new_location.clone(); diff --git a/pangolin/pangolin_store/src/memory/main.rs b/pangolin/pangolin_store/src/memory/main.rs index 39f9a94..a816faf 100644 --- a/pangolin/pangolin_store/src/memory/main.rs +++ b/pangolin/pangolin_store/src/memory/main.rs @@ -31,7 +31,15 @@ pub struct MemoryStore { pub(crate) service_users: Arc>, pub(crate) merge_operations: Arc>, pub(crate) merge_conflicts: Arc>, - pub(crate) assets_by_id: Arc, Option, String)>>, + /// Asset id -> (tenant, catalog, namespace, branch, name). + /// + /// B6: the tenant used to be absent from the value, so `delete_catalog`'s + /// index cleanup could only filter on the catalog *name* - and tenant A + /// deleting a catalog called `sales` broke `get_asset_by_id` for tenant B's + /// unrelated catalog of the same name. Catalog names are per-tenant, so the + /// index key has to be too. + pub(crate) assets_by_id: + Arc, Option, String)>>, pub(crate) revoked_tokens: Arc>, pub(crate) active_tokens: Arc>, pub(crate) system_settings: Arc>, @@ -79,3 +87,36 @@ impl MemoryStore { } } } + +/// Deterministically page an in-memory listing. +/// +/// B27: every memory-backend listing paged straight over DashMap iteration +/// order, which is a hash order that varies run to run and shifts as entries are +/// inserted or removed. Two consecutive pages could therefore repeat a row or +/// skip one entirely - the same defect the SQL backends had from `LIMIT/OFFSET` +/// with no `ORDER BY`, and it makes the "two pages cover the set exactly once" +/// property untestable. +/// +/// Sorting by a stable key before slicing gives the memory backend the same +/// observable ordering the SQL backends now get from `ORDER BY`. +pub(crate) fn paginate_sorted( + items: impl Iterator, + pagination: Option, + key: F, +) -> Vec +where + F: Fn(&T) -> K, + K: Ord, +{ + let mut all: Vec = items.collect(); + all.sort_by_key(&key); + + match pagination { + Some(p) => all + .into_iter() + .skip(p.offset.unwrap_or(0)) + .take(p.limit.unwrap_or(usize::MAX)) + .collect(), + None => all, + } +} diff --git a/pangolin/pangolin_store/src/memory/namespaces.rs b/pangolin/pangolin_store/src/memory/namespaces.rs index 4233f6f..c195719 100644 --- a/pangolin/pangolin_store/src/memory/namespaces.rs +++ b/pangolin/pangolin_store/src/memory/namespaces.rs @@ -51,13 +51,8 @@ impl MemoryStore { }) .map(|entry| entry.value().clone()); - let namespaces: Vec = if let Some(p) = pagination { - iter.skip(p.offset.unwrap_or(0)) - .take(p.limit.unwrap_or(usize::MAX)) - .collect() - } else { - iter.collect() - }; + let namespaces: Vec = + crate::memory::main::paginate_sorted(iter, pagination, |n| n.name.clone()); tracing::info!("DEBUG_MEM: Found {} namespaces", namespaces.len()); Ok(namespaces) diff --git a/pangolin/pangolin_store/src/memory/permissions.rs b/pangolin/pangolin_store/src/memory/permissions.rs index e6ddcf5..38c6155 100644 --- a/pangolin/pangolin_store/src/memory/permissions.rs +++ b/pangolin/pangolin_store/src/memory/permissions.rs @@ -26,13 +26,7 @@ impl MemoryStore { }) .map(|entry| entry.value().clone()); - let permissions = if let Some(p) = pagination { - iter.skip(p.offset.unwrap_or(0)) - .take(p.limit.unwrap_or(usize::MAX)) - .collect() - } else { - iter.collect() - }; + let permissions = crate::memory::main::paginate_sorted(iter, pagination, |p| p.id); Ok(permissions) } diff --git a/pangolin/pangolin_store/src/memory/roles.rs b/pangolin/pangolin_store/src/memory/roles.rs index fe04fb9..2a4d319 100644 --- a/pangolin/pangolin_store/src/memory/roles.rs +++ b/pangolin/pangolin_store/src/memory/roles.rs @@ -28,13 +28,7 @@ impl MemoryStore { .filter(|r| r.value().tenant_id == tenant_id) .map(|r| r.value().clone()); - let roles = if let Some(p) = pagination { - iter.skip(p.offset.unwrap_or(0)) - .take(p.limit.unwrap_or(usize::MAX)) - .collect() - } else { - iter.collect() - }; + let roles = crate::memory::main::paginate_sorted(iter, pagination, |r| r.name.clone()); Ok(roles) } pub(crate) async fn update_role_internal(&self, role: Role) -> Result<()> { diff --git a/pangolin/pangolin_store/src/memory/service_users.rs b/pangolin/pangolin_store/src/memory/service_users.rs index 2fc12d2..68c862e 100644 --- a/pangolin/pangolin_store/src/memory/service_users.rs +++ b/pangolin/pangolin_store/src/memory/service_users.rs @@ -27,13 +27,7 @@ impl MemoryStore { .filter(|entry| entry.value().tenant_id == tenant_id) .map(|entry| entry.value().clone()); - let result = if let Some(p) = pagination { - iter.skip(p.offset.unwrap_or(0)) - .take(p.limit.unwrap_or(usize::MAX)) - .collect() - } else { - iter.collect() - }; + let result = crate::memory::main::paginate_sorted(iter, pagination, |s| s.name.clone()); Ok(result) } /// Record that a service user's API key was just used. diff --git a/pangolin/pangolin_store/src/memory/tags.rs b/pangolin/pangolin_store/src/memory/tags.rs index c410f92..7965d01 100644 --- a/pangolin/pangolin_store/src/memory/tags.rs +++ b/pangolin/pangolin_store/src/memory/tags.rs @@ -42,13 +42,7 @@ impl MemoryStore { }) .map(|r| r.value().clone()); - let tags = if let Some(p) = pagination { - iter.skip(p.offset.unwrap_or(0)) - .take(p.limit.unwrap_or(usize::MAX)) - .collect() - } else { - iter.collect() - }; + let tags = crate::memory::main::paginate_sorted(iter, pagination, |t| t.name.clone()); Ok(tags) } pub(crate) async fn delete_tag_internal( diff --git a/pangolin/pangolin_store/src/memory/tenants.rs b/pangolin/pangolin_store/src/memory/tenants.rs index 15bb6c5..4d9c28b 100644 --- a/pangolin/pangolin_store/src/memory/tenants.rs +++ b/pangolin/pangolin_store/src/memory/tenants.rs @@ -53,12 +53,41 @@ impl MemoryStore { Err(anyhow::anyhow!("Tenant not found")) } } + /// Delete a tenant and everything scoped to it. + /// + /// B30: the cascade was a `// TODO`. Warehouses (with their cloud + /// credentials), catalogs, namespaces, assets, branches, tags, audit + /// history, permissions and cached tokens all survived tenant deletion on + /// the memory backend - so a "deleted" tenant's storage credentials were + /// still vendable, and a recreated tenant with the same id inherited the old + /// one's data. The retain-based pattern here is the one `delete_catalog` + /// already used. pub(crate) async fn delete_tenant_internal(&self, tenant_id: Uuid) -> Result<()> { - if self.tenants.remove(&tenant_id).is_some() { - // TODO: Cascade delete warehouses and catalogs - Ok(()) - } else { - Err(anyhow::anyhow!("Tenant not found")) + if self.tenants.remove(&tenant_id).is_none() { + return Err(anyhow::anyhow!("Tenant not found")); } + + // Keyed by (tenant, ..) - drop everything whose first key element is + // this tenant. + self.warehouses.retain(|k, _| k.0 != tenant_id); + self.catalogs.retain(|k, _| k.0 != tenant_id); + self.namespaces.retain(|k, _| k.0 != tenant_id); + self.assets.retain(|k, _| k.0 != tenant_id); + self.branches.retain(|k, _| k.0 != tenant_id); + self.tags.retain(|k, _| k.0 != tenant_id); + self.commits.retain(|k, _| k.0 != tenant_id); + self.federated_stats.retain(|k, _| k.0 != tenant_id); + + // Keyed by their own id, with the tenant in the value. + self.assets_by_id.retain(|_, v| v.0 != tenant_id); + self.audit_events.remove(&tenant_id); + self.system_settings.remove(&tenant_id); + self.users.retain(|_, u| u.tenant_id != Some(tenant_id)); + self.roles.retain(|_, r| r.tenant_id != tenant_id); + self.permissions.retain(|_, p| p.tenant_id != tenant_id); + self.service_users.retain(|_, s| s.tenant_id != tenant_id); + self.active_tokens.retain(|_, t| t.tenant_id != tenant_id); + + Ok(()) } } diff --git a/pangolin/pangolin_store/src/memory/tokens.rs b/pangolin/pangolin_store/src/memory/tokens.rs index b62ae8b..dbd73b5 100644 --- a/pangolin/pangolin_store/src/memory/tokens.rs +++ b/pangolin/pangolin_store/src/memory/tokens.rs @@ -32,13 +32,9 @@ impl MemoryStore { }) .map(|entry| entry.value().clone()); - let tokens = if let Some(p) = pagination { - iter.skip(p.offset.unwrap_or(0)) - .take(p.limit.unwrap_or(usize::MAX)) - .collect() - } else { - iter.collect() - }; + let tokens = crate::memory::main::paginate_sorted(iter, pagination, |t| { + (std::cmp::Reverse(t.expires_at), t.id) + }); Ok(tokens) } diff --git a/pangolin/pangolin_store/src/memory/users.rs b/pangolin/pangolin_store/src/memory/users.rs index 501db25..cdedf6f 100644 --- a/pangolin/pangolin_store/src/memory/users.rs +++ b/pangolin/pangolin_store/src/memory/users.rs @@ -43,13 +43,7 @@ impl MemoryStore { }) .map(|entry| entry.value().clone()); - let users = if let Some(p) = pagination { - iter.skip(p.offset.unwrap_or(0)) - .take(p.limit.unwrap_or(usize::MAX)) - .collect() - } else { - iter.collect() - }; + let users = crate::memory::main::paginate_sorted(iter, pagination, |u| u.username.clone()); Ok(users) } pub(crate) async fn update_user_internal(&self, user: User) -> Result<()> { diff --git a/pangolin/pangolin_store/src/memory/warehouses.rs b/pangolin/pangolin_store/src/memory/warehouses.rs index 3f5bcb0..f1dbbe2 100644 --- a/pangolin/pangolin_store/src/memory/warehouses.rs +++ b/pangolin/pangolin_store/src/memory/warehouses.rs @@ -36,13 +36,8 @@ impl MemoryStore { .filter(|r| r.key().0 == tenant_id) .map(|r| r.value().clone()); - let warehouses: Vec = if let Some(p) = pagination { - iter.skip(p.offset.unwrap_or(0)) - .take(p.limit.unwrap_or(usize::MAX)) - .collect() - } else { - iter.collect() - }; + let warehouses: Vec = + crate::memory::main::paginate_sorted(iter, pagination, |w| w.name.clone()); Ok(warehouses) } pub(crate) async fn update_warehouse_internal( diff --git a/pangolin/pangolin_store/src/mongo/assets.rs b/pangolin/pangolin_store/src/mongo/assets.rs index 9b0bfc1..aba0ab6 100644 --- a/pangolin/pangolin_store/src/mongo/assets.rs +++ b/pangolin/pangolin_store/src/mongo/assets.rs @@ -33,7 +33,7 @@ impl MongoStore { "namespace": namespace, "id": to_bson_uuid(asset.id), "name": &asset.name, - "kind": format!("{:?}", asset.kind), + "kind": asset.kind.as_stored_str(), "location": &asset.location, "properties": mongodb::bson::to_bson(&asset.properties)? }; @@ -72,11 +72,8 @@ impl MongoStore { if let Some(d) = doc { let kind_str = d.get_str("kind")?; - let kind = match kind_str { - "IcebergTable" => AssetType::IcebergTable, - "View" => AssetType::View, - _ => AssetType::IcebergTable, - }; + // B7: an unknown value used to fall through to `IcebergTable`. + let kind = AssetType::from_stored_str(kind_str).map_err(|e| anyhow::anyhow!(e))?; let properties: HashMap = mongodb::bson::from_bson(d.get("properties").unwrap().clone())?; @@ -128,11 +125,8 @@ impl MongoStore { let mut assets = Vec::new(); for d in docs { let kind_str = d.get_str("kind")?; - let kind = match kind_str { - "IcebergTable" => AssetType::IcebergTable, - "View" => AssetType::View, - _ => AssetType::IcebergTable, - }; + // B7: an unknown value used to fall through to `IcebergTable`. + let kind = AssetType::from_stored_str(kind_str).map_err(|e| anyhow::anyhow!(e))?; let properties: HashMap = mongodb::bson::from_bson(d.get("properties").unwrap().clone())?; @@ -180,11 +174,8 @@ impl MongoStore { .map(|v| v.as_str().unwrap().to_string()) .collect(); let kind_str = d.get_str("kind")?; - let kind = match kind_str { - "IcebergTable" => AssetType::IcebergTable, - "View" => AssetType::View, - _ => AssetType::IcebergTable, - }; + // B7: an unknown value used to fall through to `IcebergTable`. + let kind = AssetType::from_stored_str(kind_str).map_err(|e| anyhow::anyhow!(e))?; let properties: HashMap = mongodb::bson::from_bson(d.get("properties").unwrap().clone())?; diff --git a/pangolin/pangolin_store/src/mongo/audit.rs b/pangolin/pangolin_store/src/mongo/audit.rs index cd7fa0d..1aa33d1 100644 --- a/pangolin/pangolin_store/src/mongo/audit.rs +++ b/pangolin/pangolin_store/src/mongo/audit.rs @@ -6,6 +6,9 @@ use mongodb::bson::{doc, Bson, Document}; use pangolin_core::audit::{AuditLogEntry, AuditLogFilter}; use uuid::Uuid; +/// Default cap on a listing, matching the SQL backends' `LIMIT 100`. +const DEFAULT_AUDIT_LIMIT: usize = 100; + impl MongoStore { pub async fn log_audit_event(&self, entry: AuditLogEntry) -> Result<()> { let mut doc = mongodb::bson::to_document(&entry)?; @@ -31,8 +34,21 @@ impl MongoStore { Ok(()) } - pub async fn get_audit_event(&self, id: Uuid) -> Result> { - let filter = doc! { "id": to_bson_uuid(id) }; + /// Fetch one audit event, scoped to its tenant. + /// + /// B1: the filter was `{ "id": ... }` alone and the caller's `tenant_id` + /// was discarded, so any tenant holding an audit-event UUID could read + /// another tenant's audit record - username, IP, resource names, metadata. + /// Postgres and SQLite both scoped by tenant; only Mongo did not. + pub async fn get_audit_event( + &self, + tenant_id: Uuid, + id: Uuid, + ) -> Result> { + let filter = doc! { + "id": to_bson_uuid(id), + "tenant_id": to_bson_uuid(tenant_id), + }; let doc = self .db .collection::("audit_logs") @@ -60,11 +76,28 @@ impl MongoStore { tenant_id: Uuid, filter: Option, ) -> Result> { + // B23: this applied no sort, no limit and no offset while the SQL + // backends used `ORDER BY timestamp DESC LIMIT 100`. On a busy tenant + // Mongo streamed the entire audit collection into memory and returned + // it in storage order. + let (limit, offset) = filter + .as_ref() + .map(|f| { + ( + f.limit.unwrap_or(DEFAULT_AUDIT_LIMIT), + f.offset.unwrap_or(0), + ) + }) + .unwrap_or((DEFAULT_AUDIT_LIMIT, 0)); + let mongo_filter = self.build_audit_filter(tenant_id, filter)?; let cursor = self .db .collection::("audit_logs") .find(mongo_filter) + .sort(doc! { "timestamp": -1 }) + .skip(offset as u64) + .limit(limit as i64) .await?; let entries: Vec = cursor.try_collect().await?; Ok(entries) @@ -77,15 +110,29 @@ impl MongoStore { ) -> Result { let mut mongo_filter = doc! { "tenant_id": to_bson_uuid(tenant_id) }; if let Some(f) = filter { + // B23: these used `format!("{:?}", ..)` - the Debug spelling, + // `"CreateBranch"` - against documents serde wrote in snake_case + // (`"create_branch"`). The filters could never match, so an + // action- or resource-type-filtered listing always returned zero + // rows and `count_audit_events` always returned 0. Going through + // `bson::to_bson` uses the same serde naming as the write path. if let Some(rt) = f.resource_type { - mongo_filter.insert("resource_type", format!("{:?}", rt)); + mongo_filter.insert("resource_type", mongodb::bson::to_bson(&rt)?); } if let Some(ra) = f.action { - mongo_filter.insert("action", format!("{:?}", ra)); + mongo_filter.insert("action", mongodb::bson::to_bson(&ra)?); } if let Some(uid) = f.user_id { mongo_filter.insert("user_id", to_bson_uuid(uid)); } + // B23: `resource_id` and `result` were accepted by the filter type + // and then silently ignored. + if let Some(rid) = f.resource_id { + mongo_filter.insert("resource_id", to_bson_uuid(rid)); + } + if let Some(result) = f.result { + mongo_filter.insert("result", mongodb::bson::to_bson(&result)?); + } if let Some(from) = f.start_time { mongo_filter.insert("timestamp", doc! { "$gte": Bson::DateTime(from.into()) }); } diff --git a/pangolin/pangolin_store/src/mongo/main.rs b/pangolin/pangolin_store/src/mongo/main.rs index a9a2011..ece789b 100644 --- a/pangolin/pangolin_store/src/mongo/main.rs +++ b/pangolin/pangolin_store/src/mongo/main.rs @@ -203,6 +203,18 @@ impl MongoStore { } } + /// Publish a new metadata location, but only if the current one still + /// matches `expected_location`. + /// + /// B5: `expected_location` was ignored (`_expected_location`) and the update + /// was an unconditional `$set`. Memory, Postgres and SQLite all enforce the + /// compare-and-swap; on Mongo two concurrent Iceberg commits both + /// "succeeded" and one snapshot was silently lost - the exact failure class + /// the 0.6.0 work fixed at the API layer, still wide open one layer down. + /// + /// Folding the expectation into the *filter* keeps this a single-document + /// atomic update, so it works on a standalone `mongod` with no multi-document + /// transaction required. pub async fn update_metadata_location( &self, tenant_id: Uuid, @@ -210,10 +222,10 @@ impl MongoStore { branch: Option, namespace: Vec, table: String, - _expected_location: Option, + expected_location: Option, new_location: String, ) -> Result<()> { - let filter = doc! { + let mut filter = doc! { "tenant_id": to_bson_uuid(tenant_id), "catalog_name": catalog_name, "branch": branch.unwrap_or_else(|| "main".to_string()), @@ -221,16 +233,35 @@ impl MongoStore { "name": table }; + match &expected_location { + Some(expected) => { + filter.insert("properties.metadata_location", expected.clone()); + } + // `None` means "there must not be one yet" - the create-path CAS. + None => { + filter.insert("properties.metadata_location", doc! { "$exists": false }); + } + } + let update = doc! { "$set": { - "properties.metadata_location": new_location + "properties.metadata_location": &new_location, + "location": &new_location, } }; - self.db + let result = self + .db .collection::("assets") .update_one(filter, update) .await?; + + if result.matched_count == 0 { + return Err(anyhow::anyhow!( + "CAS failure: metadata location did not match {:?}", + expected_location + )); + } Ok(()) } } diff --git a/pangolin/pangolin_store/src/mongo/mod.rs b/pangolin/pangolin_store/src/mongo/mod.rs index 1fa18bc..7268555 100644 --- a/pangolin/pangolin_store/src/mongo/mod.rs +++ b/pangolin/pangolin_store/src/mongo/mod.rs @@ -471,10 +471,12 @@ impl CatalogStore for MongoStore { } async fn get_audit_event( &self, - _tenant_id: Uuid, + tenant_id: Uuid, id: Uuid, ) -> Result> { - self.get_audit_event(id).await + // B1: `tenant_id` used to be discarded here (`_tenant_id`), which is + // where the cross-tenant audit read came from. + self.get_audit_event(tenant_id, id).await } async fn count_audit_events( &self, diff --git a/pangolin/pangolin_store/src/mongo/tokens.rs b/pangolin/pangolin_store/src/mongo/tokens.rs index 2b92da7..2e80dbb 100644 --- a/pangolin/pangolin_store/src/mongo/tokens.rs +++ b/pangolin/pangolin_store/src/mongo/tokens.rs @@ -2,7 +2,7 @@ use super::main::{from_bson_uuid, to_bson_uuid}; use super::MongoStore; use anyhow::Result; use futures::stream::TryStreamExt; -use mongodb::bson::{doc, Document}; +use mongodb::bson::{doc, Bson, Document}; use pangolin_core::token::TokenInfo; use uuid::Uuid; @@ -76,16 +76,37 @@ impl MongoStore { Ok(()) } + /// Record a revocation. + /// + /// B2: this used to `insert_one(revoked)` through serde, which wrote the + /// token id as a *string* under the field name `id` (the struct's field), + /// while [`Self::is_token_revoked`] queried `token_id` as a BSON Binary + /// UUID. Neither the field name nor the type matched, so the lookup could + /// never find a revocation and the check returned `false` for every token: + /// on Mongo, revocation - including logout - was a silent no-op and revoked + /// JWTs stayed valid until they expired naturally. + /// + /// Both sides now go through an explicit `doc!` using the same encoding as + /// [`Self::store_token`]. pub async fn revoke_token( &self, token_id: Uuid, expires_at: chrono::DateTime, reason: Option, ) -> Result<()> { - let revoked = pangolin_core::token::RevokedToken::new(token_id, expires_at, reason); + let doc = doc! { + "token_id": to_bson_uuid(token_id), + "expires_at": Bson::DateTime(expires_at.into()), + "reason": reason.map(Bson::String).unwrap_or(Bson::Null), + }; + // Upsert so revoking twice is idempotent rather than accumulating rows. self.db - .collection("revoked_tokens") - .insert_one(revoked) + .collection::("revoked_tokens") + .update_one( + doc! { "token_id": to_bson_uuid(token_id) }, + doc! { "$set": doc }, + ) + .upsert(true) .await?; Ok(()) } @@ -94,18 +115,24 @@ impl MongoStore { let filter = doc! { "token_id": to_bson_uuid(token_id) }; let result = self .db - .collection::("revoked_tokens") + .collection::("revoked_tokens") .find_one(filter) .await?; Ok(result.is_some()) } + /// Drop revocation records whose tokens have expired anyway. + /// + /// B2 (second half): the comparison was `$lt` against a BSON DateTime while + /// serde had written `expires_at` as an RFC3339 *string*, so this deleted + /// nothing and the collection grew without bound. With `revoke_token` + /// writing a real `Bson::DateTime`, the comparison is now type-consistent. pub async fn cleanup_expired_tokens(&self) -> Result { let now = chrono::Utc::now(); - let filter = doc! { "expires_at": { "$lt": now } }; + let filter = doc! { "expires_at": { "$lt": Bson::DateTime(now.into()) } }; let result = self .db - .collection::("revoked_tokens") + .collection::("revoked_tokens") .delete_many(filter) .await?; Ok(result.deleted_count as usize) diff --git a/pangolin/pangolin_store/src/postgres/assets.rs b/pangolin/pangolin_store/src/postgres/assets.rs index 5e29a96..5ed4120 100644 --- a/pangolin/pangolin_store/src/postgres/assets.rs +++ b/pangolin/pangolin_store/src/postgres/assets.rs @@ -22,7 +22,7 @@ impl PostgresStore { .bind(&branch_name) .bind(&namespace) .bind(&asset.name) - .bind(format!("{:?}", asset.kind)) + .bind(asset.kind.as_stored_str()) .bind(asset.properties.get("metadata_location").unwrap_or(&asset.location)) .bind(serde_json::to_value(&asset.properties)?) .execute(&self.pool) @@ -50,11 +50,9 @@ impl PostgresStore { if let Some(row) = row { let asset_type_str: String = row.get("asset_type"); - let kind = match asset_type_str.as_str() { - "IcebergTable" => pangolin_core::model::AssetType::IcebergTable, - "View" => pangolin_core::model::AssetType::View, - _ => pangolin_core::model::AssetType::IcebergTable, - }; + // B7: an unknown value used to fall through to `IcebergTable`. + let kind = pangolin_core::model::AssetType::from_stored_str(&asset_type_str) + .map_err(|e| anyhow::anyhow!(e))?; let a = Asset { id: row.get("id"), @@ -86,11 +84,9 @@ impl PostgresStore { let catalog_name: String = row.get("catalog_name"); let namespace_path: Vec = row.get("namespace_path"); let asset_type_str: String = row.get("asset_type"); - let kind = match asset_type_str.as_str() { - "IcebergTable" => pangolin_core::model::AssetType::IcebergTable, - "View" => pangolin_core::model::AssetType::View, - _ => pangolin_core::model::AssetType::IcebergTable, - }; + // B7: an unknown value used to fall through to `IcebergTable`. + let kind = pangolin_core::model::AssetType::from_stored_str(&asset_type_str) + .map_err(|e| anyhow::anyhow!(e))?; let asset = Asset { id: row.get("id"), @@ -137,11 +133,9 @@ impl PostgresStore { let mut assets = Vec::new(); for row in rows { let asset_type_str: String = row.get("asset_type"); - let kind = match asset_type_str.as_str() { - "IcebergTable" => pangolin_core::model::AssetType::IcebergTable, - "View" => pangolin_core::model::AssetType::View, - _ => pangolin_core::model::AssetType::IcebergTable, - }; + // B7: an unknown value used to fall through to `IcebergTable`. + let kind = pangolin_core::model::AssetType::from_stored_str(&asset_type_str) + .map_err(|e| anyhow::anyhow!(e))?; let a = Asset { id: row.get("id"), diff --git a/pangolin/pangolin_store/src/postgres/branches.rs b/pangolin/pangolin_store/src/postgres/branches.rs index 26177d9..7571fc2 100644 --- a/pangolin/pangolin_store/src/postgres/branches.rs +++ b/pangolin/pangolin_store/src/postgres/branches.rs @@ -69,7 +69,7 @@ impl PostgresStore { .map(|p| p.offset.unwrap_or(0) as i64) .unwrap_or(0); - let rows = sqlx::query("SELECT name, head_commit_id, branch_type, assets FROM branches WHERE tenant_id = $1 AND catalog_name = $2 LIMIT $3 OFFSET $4") + let rows = sqlx::query("SELECT name, head_commit_id, branch_type, assets FROM branches WHERE tenant_id = $1 AND catalog_name = $2 ORDER BY name LIMIT $3 OFFSET $4") .bind(tenant_id) .bind(catalog_name) .bind(limit) diff --git a/pangolin/pangolin_store/src/postgres/catalogs.rs b/pangolin/pangolin_store/src/postgres/catalogs.rs index fd73273..6978579 100644 --- a/pangolin/pangolin_store/src/postgres/catalogs.rs +++ b/pangolin/pangolin_store/src/postgres/catalogs.rs @@ -62,7 +62,18 @@ impl PostgresStore { .map(|p| p.offset.unwrap_or(0) as i64) .unwrap_or(0); - let rows = sqlx::query("SELECT id, name, warehouse_name, storage_location, properties FROM catalogs WHERE tenant_id = $1 LIMIT $2 OFFSET $3") + // B24: the SELECT omitted `catalog_type` and `federated_config`, and the + // loop below hardcoded `Local` / `None`. Every federated catalog looked + // Local in a Postgres listing, so anything branching on `catalog_type` + // over a listing - including the federated-forwarding decision - took + // the wrong path. SQLite and Mongo returned the real values; only + // Postgres invented them. `get_catalog` two functions up decodes both + // correctly, which is the shape mirrored here. + // + // B27: `ORDER BY name` so two pages cover the set exactly once. Without + // it Postgres may return rows in any order between queries, so + // `LIMIT/OFFSET` paging could repeat or skip catalogs. + let rows = sqlx::query("SELECT id, name, catalog_type, warehouse_name, storage_location, federated_config, properties FROM catalogs WHERE tenant_id = $1 ORDER BY name LIMIT $2 OFFSET $3") .bind(tenant_id) .bind(limit) .bind(offset) @@ -71,13 +82,19 @@ impl PostgresStore { let mut catalogs = Vec::new(); for row in rows { + let catalog_type_str: String = row.get("catalog_type"); + let catalog_type = match catalog_type_str.as_str() { + "Federated" => pangolin_core::model::CatalogType::Federated, + _ => pangolin_core::model::CatalogType::Local, + }; + catalogs.push(Catalog { id: row.get("id"), name: row.get("name"), - catalog_type: pangolin_core::model::CatalogType::Local, + catalog_type, warehouse_name: row.get("warehouse_name"), storage_location: row.get("storage_location"), - federated_config: None, + federated_config: serde_json::from_value(row.get("federated_config")).ok(), properties: serde_json::from_value(row.get("properties")).unwrap_or_default(), }); } diff --git a/pangolin/pangolin_store/src/postgres/main.rs b/pangolin/pangolin_store/src/postgres/main.rs index dc6e05a..119cc9c 100644 --- a/pangolin/pangolin_store/src/postgres/main.rs +++ b/pangolin/pangolin_store/src/postgres/main.rs @@ -425,7 +425,7 @@ impl CatalogStore for PostgresStore { .unwrap_or(0); let rows = if let Some(uid) = user_id { - sqlx::query("SELECT token_id, user_id, tenant_id, token, expires_at FROM active_tokens WHERE tenant_id = $1 AND user_id = $2 AND expires_at > $3 LIMIT $4 OFFSET $5") + sqlx::query("SELECT token_id, user_id, tenant_id, token, expires_at FROM active_tokens WHERE tenant_id = $1 AND user_id = $2 AND expires_at > $3 ORDER BY expires_at DESC, token_id LIMIT $4 OFFSET $5") .bind(tenant_id) .bind(uid) .bind(Utc::now()) @@ -434,7 +434,7 @@ impl CatalogStore for PostgresStore { .fetch_all(&self.pool) .await? } else { - sqlx::query("SELECT token_id, user_id, tenant_id, token, expires_at FROM active_tokens WHERE tenant_id = $1 AND expires_at > $2 LIMIT $3 OFFSET $4") + sqlx::query("SELECT token_id, user_id, tenant_id, token, expires_at FROM active_tokens WHERE tenant_id = $1 AND expires_at > $2 ORDER BY expires_at DESC, token_id LIMIT $3 OFFSET $4") .bind(tenant_id) .bind(Utc::now()) .bind(limit) @@ -1345,10 +1345,11 @@ impl CatalogStore for PostgresStore { m.created_at as meta_created_at, m.updated_by as meta_updated_by, m.updated_at as meta_updated_at FROM assets a LEFT JOIN business_metadata m ON a.id = m.asset_id - WHERE a.tenant_id = $1 AND (a.name ILIKE $2 OR m.description ILIKE $2)" + WHERE a.tenant_id = $1 AND (a.name ILIKE $2 ESCAPE '\\' OR m.description ILIKE $2 ESCAPE '\\')" ); - let query_pattern = format!("%{}%", query); + // B28: unescaped `%`/`_` in the term were LIKE wildcards. + let query_pattern = crate::search::contains_pattern(query); let mut param_index = 3; if let Some(ref tag_list) = tags { @@ -1408,8 +1409,13 @@ impl CatalogStore for PostgresStore { }; let catalog_name: String = row.get("catalog_name"); - let namespace_path: String = row.get("namespace_path"); - let namespace: Vec = namespace_path.split('\x1F').map(String::from).collect(); + // B4: this decoded a `TEXT[]` column as `String`. `sqlx::Row::get` + // *panics* on a decode failure, so any search with at least one hit + // panicked the request - the failure only stayed hidden because a + // search with no results never reached this line. The correct + // decode is already used elsewhere in this file and in + // `postgres/assets.rs`. + let namespace: Vec = row.get("namespace_path"); results.push((asset, metadata, catalog_name, namespace)); } @@ -1418,8 +1424,9 @@ impl CatalogStore for PostgresStore { } async fn search_catalogs(&self, tenant_id: Uuid, query: &str) -> Result> { - let query_pattern = format!("%{}%", query); - let rows = sqlx::query("SELECT id, name, catalog_type, warehouse_name, storage_location, federated_config, properties FROM catalogs WHERE tenant_id = $1 AND name ILIKE $2") + // B28: unescaped `%`/`_` in the term were LIKE wildcards. + let query_pattern = crate::search::contains_pattern(query); + let rows = sqlx::query("SELECT id, name, catalog_type, warehouse_name, storage_location, federated_config, properties FROM catalogs WHERE tenant_id = $1 AND name ILIKE $2 ESCAPE '\\' ORDER BY name") .bind(tenant_id) .bind(&query_pattern) .fetch_all(&self.pool) @@ -1454,9 +1461,10 @@ impl CatalogStore for PostgresStore { tenant_id: Uuid, query: &str, ) -> Result> { - let query_pattern = format!("%{}%", query); + // B28: unescaped `%`/`_` in the term were LIKE wildcards. + let query_pattern = crate::search::contains_pattern(query); // Postgres stores namespace_path as TEXT[] - let rows = sqlx::query("SELECT catalog_name, namespace_path, properties FROM namespaces WHERE tenant_id = $1 AND array_to_string(namespace_path, '.') ILIKE $2") + let rows = sqlx::query("SELECT catalog_name, namespace_path, properties FROM namespaces WHERE tenant_id = $1 AND array_to_string(namespace_path, '.') ILIKE $2 ESCAPE '\\' ORDER BY catalog_name, namespace_path") .bind(tenant_id) .bind(&query_pattern) .fetch_all(&self.pool) @@ -1476,8 +1484,9 @@ impl CatalogStore for PostgresStore { } async fn search_branches(&self, tenant_id: Uuid, query: &str) -> Result> { - let query_pattern = format!("%{}%", query); - let rows = sqlx::query("SELECT catalog_name, name, head_commit_id, branch_type, assets FROM branches WHERE tenant_id = $1 AND name ILIKE $2") + // B28: unescaped `%`/`_` in the term were LIKE wildcards. + let query_pattern = crate::search::contains_pattern(query); + let rows = sqlx::query("SELECT catalog_name, name, head_commit_id, branch_type, assets FROM branches WHERE tenant_id = $1 AND name ILIKE $2 ESCAPE '\\' ORDER BY catalog_name, name") .bind(tenant_id) .bind(&query_pattern) .fetch_all(&self.pool) diff --git a/pangolin/pangolin_store/src/postgres/namespaces.rs b/pangolin/pangolin_store/src/postgres/namespaces.rs index 5f6956d..9fa5cf1 100644 --- a/pangolin/pangolin_store/src/postgres/namespaces.rs +++ b/pangolin/pangolin_store/src/postgres/namespaces.rs @@ -60,7 +60,7 @@ impl PostgresStore { .map(|p| p.offset.unwrap_or(0) as i64) .unwrap_or(0); - let rows = sqlx::query("SELECT namespace_path, properties FROM namespaces WHERE tenant_id = $1 AND catalog_name = $2 LIMIT $3 OFFSET $4") + let rows = sqlx::query("SELECT namespace_path, properties FROM namespaces WHERE tenant_id = $1 AND catalog_name = $2 ORDER BY namespace_path LIMIT $3 OFFSET $4") .bind(tenant_id) .bind(catalog_name) .bind(limit) diff --git a/pangolin/pangolin_store/src/postgres/permissions.rs b/pangolin/pangolin_store/src/postgres/permissions.rs index f77f87a..7899486 100644 --- a/pangolin/pangolin_store/src/postgres/permissions.rs +++ b/pangolin/pangolin_store/src/postgres/permissions.rs @@ -113,7 +113,7 @@ impl PostgresStore { "SELECT id, user_id, tenant_id, scope, actions, granted_by, granted_at FROM permissions WHERE tenant_id = $1 - LIMIT $2 OFFSET $3", + ORDER BY id LIMIT $2 OFFSET $3", ) .bind(tenant_id) .bind(limit) diff --git a/pangolin/pangolin_store/src/postgres/roles.rs b/pangolin/pangolin_store/src/postgres/roles.rs index 8727f13..b8025eb 100644 --- a/pangolin/pangolin_store/src/postgres/roles.rs +++ b/pangolin/pangolin_store/src/postgres/roles.rs @@ -57,7 +57,7 @@ impl PostgresStore { .map(|p| p.offset.unwrap_or(0) as i64) .unwrap_or(0); - let rows = sqlx::query("SELECT id, tenant_id, name, description, permissions, created_by, created_at, updated_at FROM roles WHERE tenant_id = $1 LIMIT $2 OFFSET $3") + let rows = sqlx::query("SELECT id, tenant_id, name, description, permissions, created_by, created_at, updated_at FROM roles WHERE tenant_id = $1 ORDER BY name LIMIT $2 OFFSET $3") .bind(tenant_id) .bind(limit) .bind(offset) diff --git a/pangolin/pangolin_store/src/postgres/tags.rs b/pangolin/pangolin_store/src/postgres/tags.rs index 63a4db2..521591d 100644 --- a/pangolin/pangolin_store/src/postgres/tags.rs +++ b/pangolin/pangolin_store/src/postgres/tags.rs @@ -55,7 +55,7 @@ impl PostgresStore { .map(|p| p.offset.unwrap_or(0) as i64) .unwrap_or(0); - let rows = sqlx::query("SELECT name, commit_id FROM tags WHERE tenant_id = $1 AND catalog_name = $2 LIMIT $3 OFFSET $4") + let rows = sqlx::query("SELECT name, commit_id FROM tags WHERE tenant_id = $1 AND catalog_name = $2 ORDER BY name LIMIT $3 OFFSET $4") .bind(tenant_id) .bind(catalog_name) .bind(limit) diff --git a/pangolin/pangolin_store/src/postgres/tenants.rs b/pangolin/pangolin_store/src/postgres/tenants.rs index df2a7a7..7179cdc 100644 --- a/pangolin/pangolin_store/src/postgres/tenants.rs +++ b/pangolin/pangolin_store/src/postgres/tenants.rs @@ -45,11 +45,13 @@ impl PostgresStore { .map(|p| p.offset.unwrap_or(0) as i64) .unwrap_or(0); - let rows = sqlx::query("SELECT id, name, properties FROM tenants LIMIT $1 OFFSET $2") - .bind(limit) - .bind(offset) - .fetch_all(&self.pool) - .await?; + let rows = sqlx::query( + "SELECT id, name, properties FROM tenants ORDER BY name LIMIT $1 OFFSET $2", + ) + .bind(limit) + .bind(offset) + .fetch_all(&self.pool) + .await?; let mut tenants = Vec::new(); for row in rows { diff --git a/pangolin/pangolin_store/src/postgres/users.rs b/pangolin/pangolin_store/src/postgres/users.rs index f52b27a..c5beb32 100644 --- a/pangolin/pangolin_store/src/postgres/users.rs +++ b/pangolin/pangolin_store/src/postgres/users.rs @@ -98,14 +98,14 @@ impl PostgresStore { .unwrap_or(0); let rows = if let Some(tid) = tenant_id { - sqlx::query("SELECT id, username, email, password_hash, oauth_provider, oauth_subject, tenant_id, role, active, created_at, updated_at, last_login FROM users WHERE tenant_id = $1 LIMIT $2 OFFSET $3") + sqlx::query("SELECT id, username, email, password_hash, oauth_provider, oauth_subject, tenant_id, role, active, created_at, updated_at, last_login FROM users WHERE tenant_id = $1 ORDER BY username LIMIT $2 OFFSET $3") .bind(tid) .bind(limit) .bind(offset) .fetch_all(&self.pool) .await? } else { - sqlx::query("SELECT id, username, email, password_hash, oauth_provider, oauth_subject, tenant_id, role, active, created_at, updated_at, last_login FROM users LIMIT $1 OFFSET $2") + sqlx::query("SELECT id, username, email, password_hash, oauth_provider, oauth_subject, tenant_id, role, active, created_at, updated_at, last_login FROM users ORDER BY username LIMIT $1 OFFSET $2") .bind(limit) .bind(offset) .fetch_all(&self.pool) diff --git a/pangolin/pangolin_store/src/postgres/warehouses.rs b/pangolin/pangolin_store/src/postgres/warehouses.rs index 61c72de..769b7a9 100644 --- a/pangolin/pangolin_store/src/postgres/warehouses.rs +++ b/pangolin/pangolin_store/src/postgres/warehouses.rs @@ -53,7 +53,7 @@ impl PostgresStore { .map(|p| p.offset.unwrap_or(0) as i64) .unwrap_or(0); - let rows = sqlx::query("SELECT id, name, tenant_id, use_sts, storage_config, vending_strategy FROM warehouses WHERE tenant_id = $1 LIMIT $2 OFFSET $3") + let rows = sqlx::query("SELECT id, name, tenant_id, use_sts, storage_config, vending_strategy FROM warehouses WHERE tenant_id = $1 ORDER BY name LIMIT $2 OFFSET $3") .bind(tenant_id) .bind(limit) .bind(offset) diff --git a/pangolin/pangolin_store/src/search.rs b/pangolin/pangolin_store/src/search.rs new file mode 100644 index 0000000..7e271c8 --- /dev/null +++ b/pangolin/pangolin_store/src/search.rs @@ -0,0 +1,94 @@ +//! Shared search semantics for the four backends. +//! +//! B28: search behaved four different ways. +//! +//! * **Wildcards.** Postgres and SQLite built their patterns with +//! `format!("%{}%", query)` and no escaping, so a query containing `%` or `_` +//! was interpreted as a LIKE wildcard: searching for `100%` matched +//! everything, and `a_b` matched `axb`. Mongo escaped correctly with +//! `regex::escape`, and the memory backend used a literal `contains`. Four +//! backends, four answers to the same query. +//! * **Tag filters.** Memory and SQLite matched *any* requested tag; Postgres +//! (`@>`) and Mongo (`$all`) required *all* of them. And an empty tag list +//! returned zero results on memory but everything on the others. +//! +//! This module is the single definition of both, so a backend can only diverge +//! by not calling it. + +/// The escape character used with `LIKE ... ESCAPE`. +pub const LIKE_ESCAPE_CHAR: char = '\\'; + +/// Escape LIKE metacharacters in a user-supplied search term. +/// +/// Must be paired with `ESCAPE '\'` in the SQL, which the helpers below embed. +pub fn escape_like(query: &str) -> String { + let mut escaped = String::with_capacity(query.len()); + for ch in query.chars() { + if ch == '%' || ch == '_' || ch == LIKE_ESCAPE_CHAR { + escaped.push(LIKE_ESCAPE_CHAR); + } + escaped.push(ch); + } + escaped +} + +/// Build a `%term%` LIKE pattern with metacharacters escaped. +pub fn contains_pattern(query: &str) -> String { + format!("%{}%", escape_like(query)) +} + +/// The `ESCAPE` clause every `LIKE`/`ILIKE` using [`contains_pattern`] needs. +pub const ESCAPE_CLAUSE: &str = " ESCAPE '\\'"; + +/// Does `tags` satisfy a tag filter? +/// +/// **The chosen semantic is ALL-match**: a result qualifies only if it carries +/// every requested tag. That is what Postgres's `@>` and Mongo's `$all` already +/// did, so aligning on it keeps the two SQL/document backends unchanged and +/// moves memory and SQLite - and it is the semantic faceted filtering wants, +/// where each added tag narrows the result set. +/// +/// An **empty or absent** filter means "no tag constraint", matching everything. +/// Previously an empty list returned nothing on the memory backend and +/// everything elsewhere. +pub fn tags_match(tags: &[String], required: Option<&[String]>) -> bool { + match required { + None => true, + Some(required) => required.iter().all(|want| tags.contains(want)), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn like_metacharacters_are_escaped() { + assert_eq!(escape_like("100%"), "100\\%"); + assert_eq!(escape_like("a_b"), "a\\_b"); + assert_eq!(escape_like("back\\slash"), "back\\\\slash"); + assert_eq!(escape_like("plain"), "plain"); + } + + #[test] + fn contains_pattern_wraps_the_escaped_term() { + assert_eq!(contains_pattern("50%"), "%50\\%%"); + } + + #[test] + fn tag_filter_is_all_match() { + let tags = vec!["pii".to_string(), "finance".to_string()]; + + assert!(tags_match(&tags, None), "no filter matches everything"); + assert!(tags_match(&tags, Some(&[])), "an empty filter is no filter"); + assert!(tags_match(&tags, Some(&["pii".to_string()]))); + assert!(tags_match( + &tags, + Some(&["pii".to_string(), "finance".to_string()]) + )); + assert!( + !tags_match(&tags, Some(&["pii".to_string(), "hr".to_string()])), + "every requested tag must be present" + ); + } +} diff --git a/pangolin/pangolin_store/src/sqlite/access_requests.rs b/pangolin/pangolin_store/src/sqlite/access_requests.rs index 5cc50f4..b4874dd 100644 --- a/pangolin/pangolin_store/src/sqlite/access_requests.rs +++ b/pangolin/pangolin_store/src/sqlite/access_requests.rs @@ -52,13 +52,14 @@ impl SqliteStore { .map(|p| p.offset.unwrap_or(0) as i64) .unwrap_or(0); - let rows = - sqlx::query("SELECT * FROM access_requests WHERE tenant_id = ? LIMIT ? OFFSET ?") - .bind(tenant_id.to_string()) - .bind(limit) - .bind(offset) - .fetch_all(&self.pool) - .await?; + let rows = sqlx::query( + "SELECT * FROM access_requests WHERE tenant_id = ? ORDER BY id LIMIT ? OFFSET ?", + ) + .bind(tenant_id.to_string()) + .bind(limit) + .bind(offset) + .fetch_all(&self.pool) + .await?; let mut requests = Vec::new(); for row in rows { diff --git a/pangolin/pangolin_store/src/sqlite/assets.rs b/pangolin/pangolin_store/src/sqlite/assets.rs index 9ec47d0..b4dad5f 100644 --- a/pangolin/pangolin_store/src/sqlite/assets.rs +++ b/pangolin/pangolin_store/src/sqlite/assets.rs @@ -24,7 +24,7 @@ impl SqliteStore { .bind(&namespace_path) .bind(&asset.name) .bind(&branch_name) - .bind(format!("{:?}", asset.kind)) + .bind(asset.kind.as_stored_str()) .bind(asset.properties.get("metadata_location").unwrap_or(&asset.location)) .bind(serde_json::to_string(&asset.properties)?) .execute(&self.pool) @@ -139,11 +139,9 @@ impl SqliteStore { if let Some(row) = row { let asset_type_str: String = row.get("asset_type"); - let kind = match asset_type_str.as_str() { - "IcebergTable" => AssetType::IcebergTable, - "View" => AssetType::View, - _ => AssetType::IcebergTable, - }; + // B7: an unknown value used to fall through to `IcebergTable`. + let kind = + AssetType::from_stored_str(&asset_type_str).map_err(|e| anyhow::anyhow!(e))?; Ok(Some(Asset { id: Uuid::parse_str(&row.get::("id"))?, @@ -178,11 +176,9 @@ impl SqliteStore { serde_json::from_str(&namespace_json).unwrap_or_default(); let asset_type_str: String = row.get("asset_type"); - let kind = match asset_type_str.as_str() { - "IcebergTable" => AssetType::IcebergTable, - "View" => AssetType::View, - _ => AssetType::IcebergTable, - }; + // B7: an unknown value used to fall through to `IcebergTable`. + let kind = + AssetType::from_stored_str(&asset_type_str).map_err(|e| anyhow::anyhow!(e))?; let asset = Asset { id: Uuid::parse_str(&row.get::("id"))?, @@ -218,7 +214,7 @@ impl SqliteStore { let namespace_path = serde_json::to_string(&namespace)?; let branch_name = branch.unwrap_or_else(|| "main".to_string()); - let rows = sqlx::query("SELECT id, name, asset_type, metadata_location, properties FROM assets WHERE tenant_id = ? AND catalog_name = ? AND namespace_path = ? AND branch_name = ? LIMIT ? OFFSET ?") + let rows = sqlx::query("SELECT id, name, asset_type, metadata_location, properties FROM assets WHERE tenant_id = ? AND catalog_name = ? AND namespace_path = ? AND branch_name = ? ORDER BY name LIMIT ? OFFSET ?") .bind(tenant_id.to_string()) .bind(catalog_name) .bind(&namespace_path) @@ -231,11 +227,9 @@ impl SqliteStore { let mut assets = Vec::new(); for row in rows { let asset_type_str: String = row.get("asset_type"); - let kind = match asset_type_str.as_str() { - "IcebergTable" => AssetType::IcebergTable, - "View" => AssetType::View, - _ => AssetType::IcebergTable, - }; + // B7: an unknown value used to fall through to `IcebergTable`. + let kind = + AssetType::from_stored_str(&asset_type_str).map_err(|e| anyhow::anyhow!(e))?; assets.push(Asset { id: Uuid::parse_str(&row.get::("id"))?, @@ -319,16 +313,23 @@ impl SqliteStore { &self, tenant_id: Uuid, catalog_name: &str, - _branch: Option, + branch: Option, namespace: Vec, table: String, ) -> Result> { + // B19: `branch` was discarded (`_branch`) and the query matched rows + // from *every* branch, with `fetch_optional` returning an arbitrary one. + // Reading a table on `dev` could hand back `main`'s metadata pointer - + // silently, and differently depending on row order. Postgres and Mongo + // both scope by branch. + let branch_name = branch.unwrap_or_else(|| "main".to_string()); let namespace_path = serde_json::to_string(&namespace)?; - let row = sqlx::query("SELECT metadata_location, properties FROM assets WHERE tenant_id = ? AND catalog_name = ? AND namespace_path = ? AND name = ?") + let row = sqlx::query("SELECT metadata_location, properties FROM assets WHERE tenant_id = ? AND catalog_name = ? AND namespace_path = ? AND name = ? AND branch_name = ?") .bind(tenant_id.to_string()) .bind(catalog_name) .bind(&namespace_path) .bind(&table) + .bind(&branch_name) .fetch_optional(&self.pool) .await?; @@ -388,9 +389,15 @@ impl SqliteStore { )); } - props.insert("metadata_location".to_string(), new_location); + props.insert("metadata_location".to_string(), new_location.clone()); - let update_result = sqlx::query("UPDATE assets SET properties = ? WHERE tenant_id = ? AND catalog_name = ? AND namespace_path = ? AND name = ? AND branch_name = ?") + // B20: only `properties` was updated, leaving the `metadata_location` + // *column* stale - and reads populate `Asset.location` from that + // column. On SQLite an asset's `location` was therefore frozen at + // creation time no matter how many Iceberg commits followed. + // Postgres updates both. + let update_result = sqlx::query("UPDATE assets SET metadata_location = ?, properties = ? WHERE tenant_id = ? AND catalog_name = ? AND namespace_path = ? AND name = ? AND branch_name = ?") + .bind(&new_location) .bind(serde_json::to_string(&props)?) .bind(tenant_id.to_string()) .bind(catalog_name) diff --git a/pangolin/pangolin_store/src/sqlite/audit_logs.rs b/pangolin/pangolin_store/src/sqlite/audit_logs.rs index 9e6ae43..3cd4254 100644 --- a/pangolin/pangolin_store/src/sqlite/audit_logs.rs +++ b/pangolin/pangolin_store/src/sqlite/audit_logs.rs @@ -19,14 +19,16 @@ impl SqliteStore { .bind(tenant_id.to_string()) .bind(event.user_id.map(|u| u.to_string())) .bind(&event.username) - .bind(format!("{:?}", event.action)) - .bind(format!("{:?}", event.resource_type)) + .bind(pangolin_core::audit::audit_enum_to_stored(&event.action)) + .bind(pangolin_core::audit::audit_enum_to_stored( + &event.resource_type, + )) .bind(event.resource_id.map(|u| u.to_string())) .bind(&event.resource_name) .bind(event.timestamp.timestamp_millis()) .bind(event.ip_address.as_deref().unwrap_or("")) .bind(event.user_agent.as_deref().unwrap_or("")) - .bind(format!("{:?}", event.result)) + .bind(pangolin_core::audit::audit_enum_to_stored(&event.result)) .bind(event.error_message.as_deref().unwrap_or("")) .bind(serde_json::to_string(&event.metadata)?) .execute(&self.pool) @@ -103,18 +105,22 @@ impl SqliteStore { let ts_millis: i64 = row.get("timestamp"); // Parse enums from strings + // B22: these used to lowercase the stored Debug spelling and + // deserialize against snake_case, which never matched, then swallow + // the failure with `.unwrap_or(CreateCatalog)`. Errors now + // propagate: a corrupt audit row is a loud failure, not a + // plausible-looking lie about what happened. let action_str: String = row.get("action"); - let action = serde_json::from_str(&format!("\"{}\"", action_str.to_lowercase())) - .unwrap_or(pangolin_core::audit::AuditAction::CreateCatalog); + let action = pangolin_core::audit::audit_enum_from_stored(&action_str) + .map_err(|e| anyhow::anyhow!(e))?; let resource_type_str: String = row.get("resource_type"); - let resource_type = - serde_json::from_str(&format!("\"{}\"", resource_type_str.to_lowercase())) - .unwrap_or(pangolin_core::audit::ResourceType::Catalog); + let resource_type = pangolin_core::audit::audit_enum_from_stored(&resource_type_str) + .map_err(|e| anyhow::anyhow!(e))?; let result_str: String = row.get("result"); - let result = serde_json::from_str(&format!("\"{}\"", result_str.to_lowercase())) - .unwrap_or(pangolin_core::audit::AuditResult::Success); + let result = pangolin_core::audit::audit_enum_from_stored(&result_str) + .map_err(|e| anyhow::anyhow!(e))?; events.push(AuditLogEntry { id: Uuid::parse_str(&row.get::("id"))?, @@ -183,18 +189,22 @@ impl SqliteStore { if let Some(row) = row { let ts_millis: i64 = row.get("timestamp"); + // B22: these used to lowercase the stored Debug spelling and + // deserialize against snake_case, which never matched, then swallow + // the failure with `.unwrap_or(CreateCatalog)`. Errors now + // propagate: a corrupt audit row is a loud failure, not a + // plausible-looking lie about what happened. let action_str: String = row.get("action"); - let action = serde_json::from_str(&format!("\"{}\"", action_str.to_lowercase())) - .unwrap_or(pangolin_core::audit::AuditAction::CreateCatalog); + let action = pangolin_core::audit::audit_enum_from_stored(&action_str) + .map_err(|e| anyhow::anyhow!(e))?; let resource_type_str: String = row.get("resource_type"); - let resource_type = - serde_json::from_str(&format!("\"{}\"", resource_type_str.to_lowercase())) - .unwrap_or(pangolin_core::audit::ResourceType::Catalog); + let resource_type = pangolin_core::audit::audit_enum_from_stored(&resource_type_str) + .map_err(|e| anyhow::anyhow!(e))?; let result_str: String = row.get("result"); - let result = serde_json::from_str(&format!("\"{}\"", result_str.to_lowercase())) - .unwrap_or(pangolin_core::audit::AuditResult::Success); + let result = pangolin_core::audit::audit_enum_from_stored(&result_str) + .map_err(|e| anyhow::anyhow!(e))?; Ok(Some(AuditLogEntry { id: Uuid::parse_str(&row.get::("id"))?, diff --git a/pangolin/pangolin_store/src/sqlite/branches.rs b/pangolin/pangolin_store/src/sqlite/branches.rs index bcd6e6c..4dd8d0c 100644 --- a/pangolin/pangolin_store/src/sqlite/branches.rs +++ b/pangolin/pangolin_store/src/sqlite/branches.rs @@ -71,7 +71,7 @@ impl SqliteStore { .map(|p| p.offset.unwrap_or(0) as i64) .unwrap_or(0); - let rows = sqlx::query("SELECT name, head_commit_id, branch_type, assets FROM branches WHERE tenant_id = ? AND catalog_name = ? LIMIT ? OFFSET ?") + let rows = sqlx::query("SELECT name, head_commit_id, branch_type, assets FROM branches WHERE tenant_id = ? AND catalog_name = ? ORDER BY name LIMIT ? OFFSET ?") .bind(tenant_id.to_string()) .bind(catalog_name) .bind(limit) @@ -100,19 +100,32 @@ impl SqliteStore { Ok(branches) } + /// Delete a branch and the assets that live on it. + /// + /// B3: two defects, both fatal together. The asset cleanup referenced a + /// column named `branch`, but the schema column is `branch_name` + /// (`sql/sqlite_schema.sql:71`), so the statement failed with "no such + /// column". And because the branch delete had already been committed as its + /// own statement, the branch was gone while its assets survived - orphaned + /// permanently, with no branch to reach them through - and the caller got an + /// error suggesting nothing had happened. Postgres was fixed for exactly + /// this and wraps both statements in a transaction; SQLite was never + /// patched. pub async fn delete_branch( &self, tenant_id: Uuid, catalog_name: &str, name: String, ) -> Result<()> { + let mut tx = self.pool.begin().await?; + let result = sqlx::query( "DELETE FROM branches WHERE tenant_id = ? AND catalog_name = ? AND name = ?", ) .bind(tenant_id.to_string()) .bind(catalog_name) .bind(&name) - .execute(&self.pool) + .execute(&mut *tx) .await?; if result.rows_affected() == 0 { @@ -120,13 +133,16 @@ impl SqliteStore { } // Also delete assets associated with this branch - sqlx::query("DELETE FROM assets WHERE tenant_id = ? AND catalog_name = ? AND branch = ?") - .bind(tenant_id.to_string()) - .bind(catalog_name) - .bind(&name) - .execute(&self.pool) - .await?; + sqlx::query( + "DELETE FROM assets WHERE tenant_id = ? AND catalog_name = ? AND branch_name = ?", + ) + .bind(tenant_id.to_string()) + .bind(catalog_name) + .bind(&name) + .execute(&mut *tx) + .await?; + tx.commit().await?; Ok(()) } diff --git a/pangolin/pangolin_store/src/sqlite/business_metadata.rs b/pangolin/pangolin_store/src/sqlite/business_metadata.rs index 9b8c90d..19c87e0 100644 --- a/pangolin/pangolin_store/src/sqlite/business_metadata.rs +++ b/pangolin/pangolin_store/src/sqlite/business_metadata.rs @@ -81,21 +81,28 @@ impl SqliteStore { m.created_at as meta_created_at, m.updated_by as meta_updated_by, m.updated_at as meta_updated_at FROM assets a LEFT JOIN business_metadata m ON a.id = m.asset_id - WHERE a.tenant_id = ? AND (a.name LIKE ? OR m.description LIKE ?)" + WHERE a.tenant_id = ? AND (a.name LIKE ? ESCAPE '\\' OR m.description LIKE ? ESCAPE '\\')" ); - let query_pattern = format!("%{}%", query); + let query_pattern = crate::search::contains_pattern(query); + // B28: this was an ANY-match (`EXISTS ... value IN (...)`) while + // Postgres (`@>`) and Mongo (`$all`) required *all* the requested tags. + // The chosen semantic - see `crate::search::tags_match` - is ALL-match, + // so counting distinct matches against the requested count is what makes + // SQLite agree with the other three. if let Some(ref tag_list) = tags { if !tag_list.is_empty() { - sql.push_str(" AND EXISTS (SELECT 1 FROM json_each(m.tags) WHERE value IN ("); + sql.push_str( + " AND (SELECT COUNT(DISTINCT value) FROM json_each(m.tags) WHERE value IN (", + ); for (i, _) in tag_list.iter().enumerate() { if i > 0 { sql.push_str(", "); } sql.push('?'); } - sql.push_str("))"); + sql.push_str(&format!(")) = {}", tag_list.len())); } } @@ -117,11 +124,9 @@ impl SqliteStore { for row in rows { let asset_type_str: String = row.get("asset_type"); - let kind = match asset_type_str.as_str() { - "IcebergTable" => AssetType::IcebergTable, - "View" => AssetType::View, - _ => AssetType::IcebergTable, - }; + // B7: an unknown value used to fall through to `IcebergTable`. + let kind = + AssetType::from_stored_str(&asset_type_str).map_err(|e| anyhow::anyhow!(e))?; let asset = Asset { id: Uuid::parse_str(row.get("id"))?, @@ -153,8 +158,14 @@ impl SqliteStore { }; let catalog_name: String = row.get("catalog_name"); + // B4 (SQLite sibling): `namespace_path` is stored as a JSON array + // (`serde_json::to_string(&namespace)`), so splitting it on 0x1F + // yielded a single element containing raw JSON - a search result's + // namespace came back as `["[\"a\",\"b\"]"]` rather than + // `["a", "b"]`. No panic here, just a silently wrong namespace on + // every search hit. let namespace_path: String = row.get("namespace_path"); - let namespace: Vec = namespace_path.split('\x1F').map(String::from).collect(); + let namespace: Vec = serde_json::from_str(&namespace_path).unwrap_or_default(); results.push((asset, metadata, catalog_name, namespace)); } @@ -163,8 +174,8 @@ impl SqliteStore { } pub async fn search_catalogs(&self, tenant_id: Uuid, query: &str) -> Result> { - let query_pattern = format!("%{}%", query); - let rows = sqlx::query("SELECT id, name, catalog_type, warehouse_name, storage_location, federated_config, properties FROM catalogs WHERE tenant_id = ? AND name LIKE ?") + let query_pattern = crate::search::contains_pattern(query); + let rows = sqlx::query("SELECT id, name, catalog_type, warehouse_name, storage_location, federated_config, properties FROM catalogs WHERE tenant_id = ? AND name LIKE ? ESCAPE '\\' ORDER BY name") .bind(tenant_id.to_string()) .bind(&query_pattern) .fetch_all(&self.pool) @@ -197,8 +208,8 @@ impl SqliteStore { tenant_id: Uuid, query: &str, ) -> Result> { - let query_pattern = format!("%{}%", query); - let rows = sqlx::query("SELECT catalog_name, namespace_path, properties FROM namespaces WHERE tenant_id = ? AND namespace_path LIKE ?") + let query_pattern = crate::search::contains_pattern(query); + let rows = sqlx::query("SELECT catalog_name, namespace_path, properties FROM namespaces WHERE tenant_id = ? AND namespace_path LIKE ? ESCAPE '\\' ORDER BY catalog_name, namespace_path") .bind(tenant_id.to_string()) .bind(&query_pattern) .fetch_all(&self.pool) @@ -228,8 +239,8 @@ impl SqliteStore { tenant_id: Uuid, query: &str, ) -> Result> { - let query_pattern = format!("%{}%", query); - let rows = sqlx::query("SELECT catalog_name, name, head_commit_id, branch_type, assets FROM branches WHERE tenant_id = ? AND name LIKE ?") + let query_pattern = crate::search::contains_pattern(query); + let rows = sqlx::query("SELECT catalog_name, name, head_commit_id, branch_type, assets FROM branches WHERE tenant_id = ? AND name LIKE ? ESCAPE '\\' ORDER BY catalog_name, name") .bind(tenant_id.to_string()) .bind(&query_pattern) .fetch_all(&self.pool) diff --git a/pangolin/pangolin_store/src/sqlite/catalogs.rs b/pangolin/pangolin_store/src/sqlite/catalogs.rs index b3d4c5e..3c05307 100644 --- a/pangolin/pangolin_store/src/sqlite/catalogs.rs +++ b/pangolin/pangolin_store/src/sqlite/catalogs.rs @@ -64,7 +64,7 @@ impl SqliteStore { .map(|p| p.offset.unwrap_or(0) as i64) .unwrap_or(0); - let rows = sqlx::query("SELECT id, name, catalog_type, warehouse_name, storage_location, federated_config, properties FROM catalogs WHERE tenant_id = ? LIMIT ? OFFSET ?") + let rows = sqlx::query("SELECT id, name, catalog_type, warehouse_name, storage_location, federated_config, properties FROM catalogs WHERE tenant_id = ? ORDER BY name LIMIT ? OFFSET ?") .bind(tenant_id.to_string()) .bind(limit) .bind(offset) @@ -143,48 +143,50 @@ impl SqliteStore { .ok_or_else(|| anyhow::anyhow!("Catalog not found")) } + /// Delete a catalog and cascade to its children. + /// + /// B21: this used to run five sequential deletes with no transaction, and + /// the "does the catalog exist?" check was the *last* statement. So + /// `delete_catalog(tenant, "nonexistent")` cheerfully deleted every tag, + /// branch, asset and namespace whose `catalog_name` matched - and only then + /// returned "not found", leaving the caller believing nothing had happened. + /// Any failure part-way through left the same wreckage. Postgres wraps the + /// identical cascade in a transaction; this now matches it, and checks + /// existence first. pub async fn delete_catalog(&self, tenant_id: Uuid, name: String) -> Result<()> { let tid = tenant_id.to_string(); - // Delete cascading children manually (no FK constraints on catalog_name) - // 1. Tags - sqlx::query("DELETE FROM tags WHERE tenant_id = ? AND catalog_name = ?") - .bind(&tid) - .bind(&name) - .execute(&self.pool) - .await?; - - // 2. Branches - sqlx::query("DELETE FROM branches WHERE tenant_id = ? AND catalog_name = ?") - .bind(&tid) - .bind(&name) - .execute(&self.pool) - .await?; - - // 3. Assets - sqlx::query("DELETE FROM assets WHERE tenant_id = ? AND catalog_name = ?") - .bind(&tid) - .bind(&name) - .execute(&self.pool) - .await?; + let mut tx = self.pool.begin().await?; + + // Existence first: nothing is destroyed on behalf of a catalog that is + // not there. + let exists: Option<(i64,)> = + sqlx::query_as("SELECT 1 FROM catalogs WHERE tenant_id = ? AND name = ?") + .bind(&tid) + .bind(&name) + .fetch_optional(&mut *tx) + .await?; + if exists.is_none() { + return Err(anyhow::anyhow!("Catalog '{}' not found", name)); + } - // 4. Namespaces - sqlx::query("DELETE FROM namespaces WHERE tenant_id = ? AND catalog_name = ?") - .bind(&tid) - .bind(&name) - .execute(&self.pool) - .await?; + // Delete cascading children manually (no FK constraints on catalog_name) + for table in ["tags", "branches", "assets", "namespaces"] { + let sql = format!("DELETE FROM {table} WHERE tenant_id = ? AND catalog_name = ?"); + sqlx::query(&sql) + .bind(&tid) + .bind(&name) + .execute(&mut *tx) + .await?; + } - // 5. Catalog - let result = sqlx::query("DELETE FROM catalogs WHERE tenant_id = ? AND name = ?") + sqlx::query("DELETE FROM catalogs WHERE tenant_id = ? AND name = ?") .bind(&tid) .bind(&name) - .execute(&self.pool) + .execute(&mut *tx) .await?; - if result.rows_affected() == 0 { - return Err(anyhow::anyhow!("Catalog '{}' not found", name)); - } + tx.commit().await?; Ok(()) } } diff --git a/pangolin/pangolin_store/src/sqlite/main.rs b/pangolin/pangolin_store/src/sqlite/main.rs index a51b84b..b1279f4 100644 --- a/pangolin/pangolin_store/src/sqlite/main.rs +++ b/pangolin/pangolin_store/src/sqlite/main.rs @@ -9,12 +9,15 @@ use pangolin_core::model::{SyncStats, SystemSettings}; use pangolin_core::permission::{Permission, Role, UserRole as UserRoleAssignment}; use pangolin_core::token::TokenInfo; use pangolin_core::user::{OAuthProvider, User, UserRole}; -use sqlx::sqlite::{SqlitePool, SqlitePoolOptions}; +use sqlx::sqlite::{SqliteConnectOptions, SqlitePool, SqlitePoolOptions}; use sqlx::Row; use uuid::Uuid; /// Version of `sql/sqlite_schema.sql` recorded after a successful migration. -pub const SQLITE_SCHEMA_VERSION: i64 = 1; +/// Bumped to 2: the `audit_logs` table was recreated with the columns the code +/// actually writes. Databases created before this carry the old (actor, +/// resource, details) shape, on which every audit write failed. +pub const SQLITE_SCHEMA_VERSION: i64 = 2; #[derive(Clone)] pub struct SqliteStore { @@ -49,14 +52,20 @@ impl SqliteStore { .and_then(|v| v.parse::().ok()) .unwrap_or(5); + // B18: `PRAGMA foreign_keys` is *per connection*. Running it once via + // `execute(&pool)` configured exactly one arbitrary connection out of + // the pool, so `ON DELETE CASCADE` fired or did not fire depending on + // which connection happened to serve a request - nondeterministically, + // and differently between runs. Setting it through the connect options + // means every connection in the pool is configured, including ones + // created later to grow the pool. + let connect_options = database_url + .parse::()? + .foreign_keys(true); + let pool = SqlitePoolOptions::new() .max_connections(max_connections) - .connect(database_url) - .await?; - - // Enable foreign keys - sqlx::query("PRAGMA foreign_keys = ON") - .execute(&pool) + .connect_with(connect_options) .await?; Ok(Self { @@ -114,9 +123,16 @@ impl SqliteStore { } pub async fn apply_schema(&self, schema_sql: &str) -> Result<()> { + // Schema creation runs on a single dedicated connection so the + // foreign-key toggle below is scoped to *this* work rather than + // leaking onto whichever pooled connection happened to serve it + // (B18) - the old code could leave `OFF` stuck on a connection the + // matching `ON` never touched. + let mut conn = self.pool.acquire().await?; + // Disable foreign keys during schema creation sqlx::query("PRAGMA foreign_keys = OFF") - .execute(&self.pool) + .execute(&mut *conn) .await?; // Parse statements @@ -151,12 +167,13 @@ impl SqliteStore { } for statement in statements { - sqlx::query(&statement).execute(&self.pool).await?; + sqlx::query(&statement).execute(&mut *conn).await?; } - // Re-enable foreign keys + // Re-enable foreign keys on this connection before returning it to the + // pool. sqlx::query("PRAGMA foreign_keys = ON") - .execute(&self.pool) + .execute(&mut *conn) .await?; Ok(()) } diff --git a/pangolin/pangolin_store/src/sqlite/merge_operations.rs b/pangolin/pangolin_store/src/sqlite/merge_operations.rs index 57b6bc3..92c6933 100644 --- a/pangolin/pangolin_store/src/sqlite/merge_operations.rs +++ b/pangolin/pangolin_store/src/sqlite/merge_operations.rs @@ -97,7 +97,7 @@ impl SqliteStore { let rows = sqlx::query( "SELECT id, tenant_id, catalog_name, source_branch, target_branch, base_commit_id, status, initiated_by, initiated_at, result_commit_id, completed_at - FROM merge_operations WHERE tenant_id = ? AND catalog_name = ? LIMIT ? OFFSET ?" + FROM merge_operations WHERE tenant_id = ? AND catalog_name = ? ORDER BY id LIMIT ? OFFSET ?" ) .bind(tenant_id.to_string()) .bind(catalog_name) @@ -259,7 +259,7 @@ impl SqliteStore { let rows = sqlx::query( "SELECT id, operation_id, conflict_type, asset_id, description, resolution, created_at - FROM merge_conflicts WHERE operation_id = ? LIMIT ? OFFSET ?", + FROM merge_conflicts WHERE operation_id = ? ORDER BY id LIMIT ? OFFSET ?", ) .bind(operation_id.to_string()) .bind(limit) diff --git a/pangolin/pangolin_store/src/sqlite/namespaces.rs b/pangolin/pangolin_store/src/sqlite/namespaces.rs index 99714b2..c70742c 100644 --- a/pangolin/pangolin_store/src/sqlite/namespaces.rs +++ b/pangolin/pangolin_store/src/sqlite/namespaces.rs @@ -63,7 +63,7 @@ impl SqliteStore { .map(|p| p.offset.unwrap_or(0) as i64) .unwrap_or(0); - let rows = sqlx::query("SELECT namespace_path, properties FROM namespaces WHERE tenant_id = ? AND catalog_name = ? LIMIT ? OFFSET ?") + let rows = sqlx::query("SELECT namespace_path, properties FROM namespaces WHERE tenant_id = ? AND catalog_name = ? ORDER BY namespace_path LIMIT ? OFFSET ?") .bind(tenant_id.to_string()) .bind(catalog_name) .bind(limit) diff --git a/pangolin/pangolin_store/src/sqlite/permissions.rs b/pangolin/pangolin_store/src/sqlite/permissions.rs index a9a7d0c..a7d70b3 100644 --- a/pangolin/pangolin_store/src/sqlite/permissions.rs +++ b/pangolin/pangolin_store/src/sqlite/permissions.rs @@ -114,7 +114,7 @@ impl SqliteStore { "SELECT id, user_id, tenant_id, scope, actions, granted_by, granted_at FROM permissions WHERE tenant_id = ? - LIMIT ? OFFSET ?", + ORDER BY id LIMIT ? OFFSET ?", ) .bind(tenant_id.to_string()) .bind(limit) diff --git a/pangolin/pangolin_store/src/sqlite/roles.rs b/pangolin/pangolin_store/src/sqlite/roles.rs index 868eb0f..376abad 100644 --- a/pangolin/pangolin_store/src/sqlite/roles.rs +++ b/pangolin/pangolin_store/src/sqlite/roles.rs @@ -58,7 +58,7 @@ impl SqliteStore { .map(|p| p.offset.unwrap_or(0) as i64) .unwrap_or(0); - let rows = sqlx::query("SELECT id, tenant_id, name, description, permissions, created_by, created_at, updated_at FROM roles WHERE tenant_id = ? LIMIT ? OFFSET ?") + let rows = sqlx::query("SELECT id, tenant_id, name, description, permissions, created_by, created_at, updated_at FROM roles WHERE tenant_id = ? ORDER BY name LIMIT ? OFFSET ?") .bind(tenant_id.to_string()) .bind(limit) .bind(offset) diff --git a/pangolin/pangolin_store/src/sqlite/service_users.rs b/pangolin/pangolin_store/src/sqlite/service_users.rs index d92f5ad..b745500 100644 --- a/pangolin/pangolin_store/src/sqlite/service_users.rs +++ b/pangolin/pangolin_store/src/sqlite/service_users.rs @@ -128,7 +128,7 @@ impl SqliteStore { let rows = sqlx::query( "SELECT id, name, description, tenant_id, api_key_hash, role, created_at, created_by, last_used, expires_at, active - FROM service_users WHERE tenant_id = ? LIMIT ? OFFSET ?" + FROM service_users WHERE tenant_id = ? ORDER BY name LIMIT ? OFFSET ?" ) .bind(tenant_id.to_string()) .bind(limit) diff --git a/pangolin/pangolin_store/src/sqlite/tags.rs b/pangolin/pangolin_store/src/sqlite/tags.rs index 346081d..906376b 100644 --- a/pangolin/pangolin_store/src/sqlite/tags.rs +++ b/pangolin/pangolin_store/src/sqlite/tags.rs @@ -55,7 +55,7 @@ impl SqliteStore { .map(|p| p.offset.unwrap_or(0) as i64) .unwrap_or(0); - let rows = sqlx::query("SELECT name, commit_id FROM tags WHERE tenant_id = ? AND catalog_name = ? LIMIT ? OFFSET ?") + let rows = sqlx::query("SELECT name, commit_id FROM tags WHERE tenant_id = ? AND catalog_name = ? ORDER BY name LIMIT ? OFFSET ?") .bind(tenant_id.to_string()) .bind(catalog_name) .bind(limit) diff --git a/pangolin/pangolin_store/src/sqlite/tenants.rs b/pangolin/pangolin_store/src/sqlite/tenants.rs index 0188c91..83995de 100644 --- a/pangolin/pangolin_store/src/sqlite/tenants.rs +++ b/pangolin/pangolin_store/src/sqlite/tenants.rs @@ -44,11 +44,12 @@ impl SqliteStore { .map(|p| p.offset.unwrap_or(0) as i64) .unwrap_or(0); - let rows = sqlx::query("SELECT id, name, properties FROM tenants LIMIT ? OFFSET ?") - .bind(limit) - .bind(offset) - .fetch_all(&self.pool) - .await?; + let rows = + sqlx::query("SELECT id, name, properties FROM tenants ORDER BY name LIMIT ? OFFSET ?") + .bind(limit) + .bind(offset) + .fetch_all(&self.pool) + .await?; let mut tenants = Vec::new(); for row in rows { diff --git a/pangolin/pangolin_store/src/sqlite/tokens.rs b/pangolin/pangolin_store/src/sqlite/tokens.rs index f091e98..67e2ffd 100644 --- a/pangolin/pangolin_store/src/sqlite/tokens.rs +++ b/pangolin/pangolin_store/src/sqlite/tokens.rs @@ -7,6 +7,56 @@ use sqlx::Row; use uuid::Uuid; impl SqliteStore { + /// Record a token revocation. + /// + /// Found by the cross-backend parity suite: `SqliteStore` had **no** + /// inherent `revoke_token`, `is_token_revoked` or `cleanup_expired_tokens`, + /// so the trait implementations in `sqlite/main.rs` - written as + /// `self.revoke_token(..)` in the style of every other delegation in that + /// file - resolved to the *trait* method and called themselves. On SQLite, + /// revoking a token (i.e. logging out) recursed until the thread's stack was + /// exhausted and aborted the process: an unauthenticated-adjacent remote + /// crash, not merely a missing feature. + /// + /// The `revoked_tokens` table has existed in the schema since A-27; only the + /// code to use it was missing. + pub async fn revoke_token( + &self, + token_id: Uuid, + expires_at: chrono::DateTime, + reason: Option, + ) -> Result<()> { + sqlx::query( + "INSERT INTO revoked_tokens (token_id, expires_at, reason) VALUES (?, ?, ?) + ON CONFLICT(token_id) DO UPDATE SET + expires_at = excluded.expires_at, + reason = excluded.reason", + ) + .bind(token_id.to_string()) + .bind(expires_at.timestamp_millis()) + .bind(reason) + .execute(&self.pool) + .await?; + Ok(()) + } + + pub async fn is_token_revoked(&self, token_id: Uuid) -> Result { + let row: Option<(i64,)> = sqlx::query_as("SELECT 1 FROM revoked_tokens WHERE token_id = ?") + .bind(token_id.to_string()) + .fetch_optional(&self.pool) + .await?; + Ok(row.is_some()) + } + + /// Drop revocation records whose tokens have expired anyway. + pub async fn cleanup_expired_tokens(&self) -> Result { + let result = sqlx::query("DELETE FROM revoked_tokens WHERE expires_at < ?") + .bind(Utc::now().timestamp_millis()) + .execute(&self.pool) + .await?; + Ok(result.rows_affected() as usize) + } + pub async fn store_token(&self, token_info: TokenInfo) -> Result<()> { sqlx::query("INSERT INTO active_tokens (token_id, user_id, tenant_id, token, expires_at, created_at) VALUES (?, ?, ?, ?, ?, ?)") .bind(token_info.id.to_string()) @@ -76,7 +126,7 @@ impl SqliteStore { .unwrap_or(0); let rows = if let Some(uid) = user_id { - sqlx::query("SELECT token_id, user_id, tenant_id, token, expires_at FROM active_tokens WHERE tenant_id = ? AND user_id = ? AND expires_at > ? LIMIT ? OFFSET ?") + sqlx::query("SELECT token_id, user_id, tenant_id, token, expires_at FROM active_tokens WHERE tenant_id = ? AND user_id = ? AND expires_at > ? ORDER BY expires_at DESC, token_id LIMIT ? OFFSET ?") .bind(tenant_id.to_string()) .bind(uid.to_string()) .bind(Utc::now().timestamp()) @@ -85,7 +135,7 @@ impl SqliteStore { .fetch_all(&self.pool) .await? } else { - sqlx::query("SELECT token_id, user_id, tenant_id, token, expires_at FROM active_tokens WHERE tenant_id = ? AND expires_at > ? LIMIT ? OFFSET ?") + sqlx::query("SELECT token_id, user_id, tenant_id, token, expires_at FROM active_tokens WHERE tenant_id = ? AND expires_at > ? ORDER BY expires_at DESC, token_id LIMIT ? OFFSET ?") .bind(tenant_id.to_string()) .bind(Utc::now().timestamp()) .bind(limit) diff --git a/pangolin/pangolin_store/src/sqlite/users.rs b/pangolin/pangolin_store/src/sqlite/users.rs index 8164d41..013fd3e 100644 --- a/pangolin/pangolin_store/src/sqlite/users.rs +++ b/pangolin/pangolin_store/src/sqlite/users.rs @@ -56,14 +56,14 @@ impl SqliteStore { .unwrap_or(0); let rows = if let Some(tid) = tenant_id { - sqlx::query("SELECT id, username, email, password_hash, oauth_provider, oauth_subject, tenant_id, role, created_at, updated_at, last_login, active FROM users WHERE tenant_id = ? LIMIT ? OFFSET ?") + sqlx::query("SELECT id, username, email, password_hash, oauth_provider, oauth_subject, tenant_id, role, created_at, updated_at, last_login, active FROM users WHERE tenant_id = ? ORDER BY username LIMIT ? OFFSET ?") .bind(tid.to_string()) .bind(limit) .bind(offset) .fetch_all(&self.pool) .await? } else { - sqlx::query("SELECT id, username, email, password_hash, oauth_provider, oauth_subject, tenant_id, role, created_at, updated_at, last_login, active FROM users LIMIT ? OFFSET ?") + sqlx::query("SELECT id, username, email, password_hash, oauth_provider, oauth_subject, tenant_id, role, created_at, updated_at, last_login, active FROM users ORDER BY username LIMIT ? OFFSET ?") .bind(limit) .bind(offset) .fetch_all(&self.pool) diff --git a/pangolin/pangolin_store/src/sqlite/warehouses.rs b/pangolin/pangolin_store/src/sqlite/warehouses.rs index 78aa470..78f7ff6 100644 --- a/pangolin/pangolin_store/src/sqlite/warehouses.rs +++ b/pangolin/pangolin_store/src/sqlite/warehouses.rs @@ -55,7 +55,7 @@ impl SqliteStore { .map(|p| p.offset.unwrap_or(0) as i64) .unwrap_or(0); - let rows = sqlx::query("SELECT id, name, use_sts, storage_config, vending_strategy FROM warehouses WHERE tenant_id = ? LIMIT ? OFFSET ?") + let rows = sqlx::query("SELECT id, name, use_sts, storage_config, vending_strategy FROM warehouses WHERE tenant_id = ? ORDER BY name LIMIT ? OFFSET ?") .bind(tenant_id.to_string()) .bind(limit) .bind(offset) diff --git a/pangolin/pangolin_store/src/tests/mod.rs b/pangolin/pangolin_store/src/tests/mod.rs index bceed31..f06b9a3 100644 --- a/pangolin/pangolin_store/src/tests/mod.rs +++ b/pangolin/pangolin_store/src/tests/mod.rs @@ -273,6 +273,8 @@ pub async fn test_dashboard_stats_consistency(store: &S) { pub mod audit_tests; #[cfg(test)] pub mod multi_cloud; +/// Cross-backend parity suite (roadmap improvement #1). +pub mod parity; #[cfg(test)] pub mod postgres_merge_tests; #[cfg(test)] diff --git a/pangolin/pangolin_store/src/tests/parity.rs b/pangolin/pangolin_store/src/tests/parity.rs new file mode 100644 index 0000000..146a429 --- /dev/null +++ b/pangolin/pangolin_store/src/tests/parity.rs @@ -0,0 +1,764 @@ +//! Cross-backend parity suite. +//! +//! Roadmap improvement #1. Nearly half the storage-layer findings in the +//! August audit (B1-B7, B17-B30) were one backend silently diverging from the +//! other three: a tenant filter dropped on Mongo, a CAS skipped in memory, an +//! enum round-tripping to the wrong variant on all the SQL backends, pagination +//! that repeats or skips rows, four different answers to the same search. +//! +//! Each of those is invisible to a per-backend test, because per-backend tests +//! assert what that backend does. What catches them is asserting that all four +//! backends do the *same* thing. Every function here runs against whichever +//! `CatalogStore` it is handed, and `tests/store_integration.rs` runs the whole +//! set against memory, SQLite, Postgres and Mongo. +//! +//! Each assertion names the finding it locks down, so a regression points +//! straight at what it broke. + +use crate::CatalogStore; +use pangolin_core::business_metadata::BusinessMetadata; +use pangolin_core::model::*; +use std::collections::HashMap; +use uuid::Uuid; + +/// Build the tenant -> warehouse -> catalog -> namespace chain the SQL backends' +/// foreign keys require, and return the tenant id. +async fn seed_hierarchy(store: &S, catalog: &str, namespace: &[String]) -> Uuid { + let tenant_id = Uuid::new_v4(); + + store + .create_tenant(Tenant { + id: tenant_id, + name: format!("parity_{tenant_id}"), + properties: HashMap::new(), + }) + .await + .expect("create tenant"); + + let _ = store + .create_warehouse( + tenant_id, + Warehouse { + id: Uuid::new_v4(), + name: "wh".to_string(), + tenant_id, + storage_config: HashMap::new(), + use_sts: false, + vending_strategy: None, + }, + ) + .await; + + store + .create_catalog( + tenant_id, + Catalog { + id: Uuid::new_v4(), + name: catalog.to_string(), + catalog_type: CatalogType::Local, + warehouse_name: Some("wh".to_string()), + storage_location: None, + federated_config: None, + properties: HashMap::new(), + }, + ) + .await + .expect("create catalog"); + + store + .create_namespace( + tenant_id, + catalog, + Namespace { + name: namespace.to_vec(), + properties: HashMap::new(), + }, + ) + .await + .expect("create namespace"); + + tenant_id +} + +fn asset(name: &str, kind: AssetType) -> Asset { + Asset { + id: Uuid::new_v4(), + name: name.to_string(), + kind, + location: format!("s3://bucket/{name}"), + properties: HashMap::new(), + } +} + +/// **B7.** Every `AssetType` variant must survive a write/read round trip. +/// +/// All three persistent backends stored the Debug spelling and parsed only +/// `IcebergTable`/`View`, defaulting the other 15 variants to `IcebergTable` - +/// so a `DeltaTable` came back as an Iceberg table with no error anywhere. +pub async fn asset_types_round_trip(store: &S) { + let catalog = "types"; + let namespace = vec!["ns".to_string()]; + let tenant_id = seed_hierarchy(store, catalog, &namespace).await; + + for kind in AssetType::all() { + let name = format!("asset_{}", kind.as_stored_str().to_lowercase()); + let written = asset(&name, kind.clone()); + + store + .create_asset(tenant_id, catalog, None, namespace.clone(), written.clone()) + .await + .expect("create asset"); + + let read = store + .get_asset(tenant_id, catalog, None, namespace.clone(), name.clone()) + .await + .expect("get asset") + .unwrap_or_else(|| panic!("asset {name} vanished")); + + assert_eq!( + read.kind, kind, + "asset type {kind:?} did not round-trip (B7); it came back as {:?}", + read.kind + ); + } +} + +/// **B27.** Two pages must cover the set exactly once. +/// +/// Every paginated query outside three Postgres call sites ran `LIMIT/OFFSET` +/// with no `ORDER BY`, and the memory backend paged over DashMap iteration +/// order. Both can repeat or skip a row between pages, which is invisible until +/// a client silently misses data. +pub async fn pagination_covers_the_set_exactly_once(store: &S) { + let catalog = "paging"; + let namespace = vec!["ns".to_string()]; + let tenant_id = seed_hierarchy(store, catalog, &namespace).await; + + const TOTAL: usize = 7; + const PAGE: usize = 3; + + for i in 0..TOTAL { + store + .create_asset( + tenant_id, + catalog, + None, + namespace.clone(), + // Zero-padded so lexical and numeric order agree, and the + // assertion is about paging rather than about collation. + asset(&format!("t{i:03}"), AssetType::IcebergTable), + ) + .await + .expect("create asset"); + } + + let mut seen: Vec = Vec::new(); + let mut offset = 0; + loop { + let page = store + .list_assets( + tenant_id, + catalog, + None, + namespace.clone(), + Some(crate::PaginationParams { + limit: Some(PAGE), + offset: Some(offset), + }), + ) + .await + .expect("list assets"); + + if page.is_empty() { + break; + } + seen.extend(page.iter().map(|a| a.name.clone())); + offset += PAGE; + + // Guard against a backend that ignores offset and loops forever. + assert!(offset <= TOTAL * 4, "pagination did not terminate"); + } + + let mut unique = seen.clone(); + unique.sort(); + unique.dedup(); + + assert_eq!( + unique.len(), + seen.len(), + "paging returned a duplicate row (B27): {seen:?}" + ); + assert_eq!( + unique.len(), + TOTAL, + "paging skipped a row (B27): saw {} of {TOTAL}", + unique.len() + ); +} + +/// **B27.** Repeating the same listing must return the same order. +pub async fn listing_order_is_stable(store: &S) { + let catalog = "stable"; + let namespace = vec!["ns".to_string()]; + let tenant_id = seed_hierarchy(store, catalog, &namespace).await; + + for name in ["delta", "alpha", "charlie", "bravo"] { + store + .create_asset( + tenant_id, + catalog, + None, + namespace.clone(), + asset(name, AssetType::IcebergTable), + ) + .await + .expect("create asset"); + } + + let first: Vec = store + .list_assets(tenant_id, catalog, None, namespace.clone(), None) + .await + .expect("list assets") + .into_iter() + .map(|a| a.name) + .collect(); + + for _ in 0..3 { + let again: Vec = store + .list_assets(tenant_id, catalog, None, namespace.clone(), None) + .await + .expect("list assets") + .into_iter() + .map(|a| a.name) + .collect(); + assert_eq!(again, first, "listing order is not stable (B27)"); + } +} + +/// **B2 / B0j.** A revoked token must read back as revoked. +/// +/// On Mongo the revocation write and the revocation *check* used different +/// field names and different types, so the check could never match: revocation +/// - including logout - was a silent no-op. +pub async fn revocation_round_trips(store: &S) { + let token_id = Uuid::new_v4(); + let expires_at = chrono::Utc::now() + chrono::Duration::hours(1); + + assert!( + !store + .is_token_revoked(token_id) + .await + .expect("check revocation"), + "a fresh token must not be revoked" + ); + + store + .revoke_token(token_id, expires_at, Some("parity test".to_string())) + .await + .expect("revoke token"); + + assert!( + store + .is_token_revoked(token_id) + .await + .expect("check revocation"), + "a revoked token must read back as revoked (B2)" + ); +} + +/// **B1.** An audit event must not be readable from another tenant. +pub async fn audit_events_are_tenant_scoped(store: &S) { + let owner = Uuid::new_v4(); + let stranger = Uuid::new_v4(); + + for tenant in [owner, stranger] { + store + .create_tenant(Tenant { + id: tenant, + name: format!("audit_{tenant}"), + properties: HashMap::new(), + }) + .await + .expect("create tenant"); + } + + let entry = pangolin_core::audit::AuditLogEntry::success( + owner, + Some(Uuid::new_v4()), + "owner".to_string(), + pangolin_core::audit::AuditAction::CreateCatalog, + pangolin_core::audit::ResourceType::Catalog, + Some(Uuid::new_v4()), + "secret_catalog".to_string(), + ); + let event_id = entry.id; + + store + .log_audit_event(owner, entry) + .await + .expect("log audit event"); + + assert!( + store + .get_audit_event(owner, event_id) + .await + .expect("get audit event") + .is_some(), + "the owning tenant must be able to read its own audit event" + ); + + assert!( + store + .get_audit_event(stranger, event_id) + .await + .expect("get audit event") + .is_none(), + "an audit event must not be readable across tenants (B1)" + ); +} + +/// **B22 / B23.** Audit actions must round-trip, not collapse to a default. +/// +/// SQLite persisted the Debug spelling and parsed snake_case, then swallowed the +/// mismatch with `unwrap_or(CreateCatalog)`, so nearly every multi-word action +/// was misattributed. Mongo's filters had the mirror-image problem and always +/// matched zero rows. +pub async fn audit_actions_round_trip(store: &S) { + use pangolin_core::audit::{AuditAction, AuditLogEntry, ResourceType}; + + let tenant_id = Uuid::new_v4(); + store + .create_tenant(Tenant { + id: tenant_id, + name: format!("audit_actions_{tenant_id}"), + properties: HashMap::new(), + }) + .await + .expect("create tenant"); + + // Deliberately multi-word: the single-word ones happened to survive. + let actions = [ + AuditAction::CreateBranch, + AuditAction::DeleteNamespace, + AuditAction::CommitTable, + ]; + + for action in &actions { + store + .log_audit_event( + tenant_id, + AuditLogEntry::success( + tenant_id, + Some(Uuid::new_v4()), + "auditor".to_string(), + action.clone(), + ResourceType::Table, + Some(Uuid::new_v4()), + format!("{action:?}_target"), + ), + ) + .await + .expect("log audit event"); + } + + let events = store + .list_audit_events(tenant_id, None) + .await + .expect("list audit events"); + + for action in &actions { + assert!( + events.iter().any(|e| e.action == *action), + "audit action {action:?} did not round-trip (B22); \ + the listing held {:?}", + events.iter().map(|e| &e.action).collect::>() + ); + } +} + +/// **B28.** A search term containing LIKE metacharacters must be literal. +pub async fn search_treats_wildcards_literally(store: &S) { + let catalog = "search"; + let namespace = vec!["ns".to_string()]; + let tenant_id = seed_hierarchy(store, catalog, &namespace).await; + + for name in ["margin_100pct", "unrelated_table"] { + store + .create_asset( + tenant_id, + catalog, + None, + namespace.clone(), + asset(name, AssetType::IcebergTable), + ) + .await + .expect("create asset"); + } + + // `%` is a LIKE wildcard. Unescaped, this matched everything. + let hits = store + .search_assets(tenant_id, "%", None) + .await + .expect("search assets"); + + assert!( + hits.is_empty(), + "a literal '%' matched {} assets (B28); the term was treated as a wildcard", + hits.len() + ); + + // A genuine substring still matches. + let hits = store + .search_assets(tenant_id, "margin", None) + .await + .expect("search assets"); + assert_eq!( + hits.len(), + 1, + "an ordinary substring search should still find its asset" + ); +} + +/// **B28.** Tag filtering is ALL-match, and an empty filter is no filter. +pub async fn tag_filter_semantics_agree(store: &S) { + let catalog = "tags"; + let namespace = vec!["ns".to_string()]; + let tenant_id = seed_hierarchy(store, catalog, &namespace).await; + + let both = asset("tagged_both", AssetType::IcebergTable); + let one = asset("tagged_one", AssetType::IcebergTable); + + for a in [&both, &one] { + store + .create_asset(tenant_id, catalog, None, namespace.clone(), a.clone()) + .await + .expect("create asset"); + } + + let mut meta_both = BusinessMetadata::new(both.id, Uuid::new_v4()); + meta_both.tags = vec!["pii".to_string(), "finance".to_string()]; + store + .upsert_business_metadata(meta_both) + .await + .expect("upsert metadata"); + + let mut meta_one = BusinessMetadata::new(one.id, Uuid::new_v4()); + meta_one.tags = vec!["pii".to_string()]; + store + .upsert_business_metadata(meta_one) + .await + .expect("upsert metadata"); + + // One tag: both assets carry it. + let hits = store + .search_assets(tenant_id, "tagged", Some(vec!["pii".to_string()])) + .await + .expect("search assets"); + assert_eq!(hits.len(), 2, "single-tag filter should match both assets"); + + // Two tags: ALL-match, so only the asset carrying both. + let hits = store + .search_assets( + tenant_id, + "tagged", + Some(vec!["pii".to_string(), "finance".to_string()]), + ) + .await + .expect("search assets"); + assert_eq!( + hits.len(), + 1, + "the tag filter must be ALL-match (B28): every requested tag has to be present" + ); + + // An empty filter is not a filter. + let hits = store + .search_assets(tenant_id, "tagged", Some(vec![])) + .await + .expect("search assets"); + assert_eq!( + hits.len(), + 2, + "an empty tag list must mean 'no tag filter' (B28)" + ); +} + +/// **B26 / B5.** The compare-and-swap must actually compare. +/// +/// Mongo ignored `expected_location` entirely, so two concurrent commits both +/// "succeeded" and one snapshot was lost; memory skipped the check whenever the +/// expectation was `None`. +pub async fn metadata_cas_rejects_a_stale_writer(store: &S) { + let catalog = "cas"; + let namespace = vec!["ns".to_string()]; + let tenant_id = seed_hierarchy(store, catalog, &namespace).await; + + let table = "committed".to_string(); + let v1 = "s3://bucket/cas/v1.json".to_string(); + let mut a = asset(&table, AssetType::IcebergTable); + a.location = v1.clone(); + a.properties + .insert("metadata_location".to_string(), v1.clone()); + + store + .create_asset(tenant_id, catalog, None, namespace.clone(), a) + .await + .expect("create asset"); + + // Writer A wins. + let v2 = "s3://bucket/cas/v2.json".to_string(); + store + .update_metadata_location( + tenant_id, + catalog, + None, + namespace.clone(), + table.clone(), + Some(v1.clone()), + v2.clone(), + ) + .await + .expect("the first writer's CAS should succeed"); + + // Writer B still believes the table is at v1: it must be refused. + let v3 = "s3://bucket/cas/v3.json".to_string(); + let stale = store + .update_metadata_location( + tenant_id, + catalog, + None, + namespace.clone(), + table.clone(), + Some(v1.clone()), + v3, + ) + .await; + + assert!( + stale.is_err(), + "a stale writer's CAS must fail (B5/B26); it silently overwrote the winner" + ); + + let current = store + .get_metadata_location(tenant_id, catalog, None, namespace, table) + .await + .expect("get metadata location"); + assert_eq!( + current, + Some(v2), + "the losing writer must not have changed the published metadata" + ); +} + +/// **B6 / B30.** Deleting one tenant's data must not disturb another's. +/// +/// Catalog names are per-tenant, but the memory backend's by-id asset index was +/// keyed on catalog name alone, so deleting tenant A's `sales` broke +/// `get_asset_by_id` for tenant B's unrelated `sales`. +pub async fn deleting_a_catalog_does_not_touch_another_tenant(store: &S) { + let catalog = "sales"; + let namespace = vec!["ns".to_string()]; + + let tenant_a = seed_hierarchy(store, catalog, &namespace).await; + let tenant_b = seed_hierarchy(store, catalog, &namespace).await; + + let a_asset = asset("orders", AssetType::IcebergTable); + let b_asset = asset("orders", AssetType::IcebergTable); + + store + .create_asset(tenant_a, catalog, None, namespace.clone(), a_asset.clone()) + .await + .expect("create asset"); + store + .create_asset(tenant_b, catalog, None, namespace.clone(), b_asset.clone()) + .await + .expect("create asset"); + + store + .delete_catalog(tenant_a, catalog.to_string()) + .await + .expect("delete catalog"); + + let survivor = store + .get_asset_by_id(tenant_b, b_asset.id) + .await + .expect("get asset by id"); + + assert!( + survivor.is_some(), + "deleting tenant A's catalog must not evict tenant B's identically-named \ + catalog from the by-id index (B6)" + ); +} + +/// **B21.** Deleting a catalog that does not exist must destroy nothing. +pub async fn deleting_a_missing_catalog_destroys_nothing(store: &S) { + let catalog = "present"; + let namespace = vec!["ns".to_string()]; + let tenant_id = seed_hierarchy(store, catalog, &namespace).await; + + let kept = asset("keep_me", AssetType::IcebergTable); + store + .create_asset(tenant_id, catalog, None, namespace.clone(), kept.clone()) + .await + .expect("create asset"); + + // The SQLite cascade used to run *before* the existence check, so this call + // deleted every child row whose catalog_name matched and only then errored. + let result = store.delete_catalog(tenant_id, "absent".to_string()).await; + assert!( + result.is_err(), + "deleting a nonexistent catalog should be an error" + ); + + let still_there = store + .get_asset(tenant_id, catalog, None, namespace, kept.name.clone()) + .await + .expect("get asset"); + assert!( + still_there.is_some(), + "a failed delete_catalog must not have destroyed another catalog's assets (B21)" + ); +} + +/// **B16h.** Namespace property removals must actually remove. +pub async fn namespace_property_replacement_removes_keys(store: &S) { + let catalog = "props"; + let namespace = vec!["ns".to_string()]; + let tenant_id = seed_hierarchy(store, catalog, &namespace).await; + + let mut properties = HashMap::new(); + properties.insert("keep".to_string(), "yes".to_string()); + properties.insert("drop".to_string(), "please".to_string()); + + store + .update_namespace_properties(tenant_id, catalog, namespace.clone(), properties) + .await + .expect("seed properties"); + + let mut remaining = HashMap::new(); + remaining.insert("keep".to_string(), "yes".to_string()); + + store + .replace_namespace_properties(tenant_id, catalog, namespace.clone(), remaining) + .await + .expect("replace properties"); + + let ns = store + .get_namespace(tenant_id, catalog, namespace) + .await + .expect("get namespace") + .expect("namespace should exist"); + + assert_eq!(ns.properties.get("keep").map(String::as_str), Some("yes")); + assert!( + !ns.properties.contains_key("drop"), + "replace_namespace_properties must drop keys the caller left out (B16h)" + ); +} + +/// **B17 / B19.** A multi-level namespace must be usable, not just creatable. +pub async fn nested_namespaces_are_addressable(store: &S) { + let catalog = "nested"; + let namespace = vec!["outer".to_string(), "inner".to_string()]; + let tenant_id = seed_hierarchy(store, catalog, &namespace).await; + + let found = store + .get_namespace(tenant_id, catalog, namespace.clone()) + .await + .expect("get namespace"); + assert!( + found.is_some(), + "a nested namespace must be retrievable by its full path (B17)" + ); + + let mut properties = HashMap::new(); + properties.insert("level".to_string(), "two".to_string()); + store + .update_namespace_properties(tenant_id, catalog, namespace.clone(), properties) + .await + .expect("a nested namespace must be updatable (B17)"); + + store + .delete_namespace(tenant_id, catalog, namespace.clone()) + .await + .expect("a nested namespace must be deletable (B17)"); + + assert!( + store + .get_namespace(tenant_id, catalog, namespace) + .await + .expect("get namespace") + .is_none(), + "the namespace should be gone after delete" + ); +} + +/// **B24.** A federated catalog must still look federated in a *listing*. +pub async fn catalog_type_survives_a_listing(store: &S) { + let tenant_id = Uuid::new_v4(); + store + .create_tenant(Tenant { + id: tenant_id, + name: format!("fed_{tenant_id}"), + properties: HashMap::new(), + }) + .await + .expect("create tenant"); + + store + .create_catalog( + tenant_id, + Catalog { + id: Uuid::new_v4(), + name: "remote".to_string(), + catalog_type: CatalogType::Federated, + warehouse_name: None, + storage_location: None, + federated_config: Some(FederatedCatalogConfig { + properties: HashMap::new(), + }), + properties: HashMap::new(), + }, + ) + .await + .expect("create catalog"); + + let listed = store + .list_catalogs(tenant_id, None) + .await + .expect("list catalogs"); + + let remote = listed + .iter() + .find(|c| c.name == "remote") + .expect("the federated catalog should be listed"); + + assert_eq!( + remote.catalog_type, + CatalogType::Federated, + "list_catalogs must report the real catalog type (B24); \ + Postgres used to hardcode Local" + ); +} + +/// Run the whole parity suite against one backend. +/// +/// `tests/store_integration.rs` calls this for each of the four, which is what +/// turns "this backend behaves like this" into "all four behave alike". +pub async fn run_all(store: &S) { + asset_types_round_trip(store).await; + pagination_covers_the_set_exactly_once(store).await; + listing_order_is_stable(store).await; + revocation_round_trips(store).await; + audit_events_are_tenant_scoped(store).await; + audit_actions_round_trip(store).await; + search_treats_wildcards_literally(store).await; + tag_filter_semantics_agree(store).await; + metadata_cas_rejects_a_stale_writer(store).await; + deleting_a_catalog_does_not_touch_another_tenant(store).await; + deleting_a_missing_catalog_destroys_nothing(store).await; + namespace_property_replacement_removes_keys(store).await; + nested_namespaces_are_addressable(store).await; + catalog_type_survives_a_listing(store).await; +} diff --git a/pangolin/pangolin_store/tests/mongo_audit_tests.rs b/pangolin/pangolin_store/tests/mongo_audit_tests.rs index 477890e..8f1af15 100644 --- a/pangolin/pangolin_store/tests/mongo_audit_tests.rs +++ b/pangolin/pangolin_store/tests/mongo_audit_tests.rs @@ -196,11 +196,21 @@ async fn test_mongo_audit_log_filtering() { // Test 10: Get individual event let event_id = logs[0].id; - let event = store.get_audit_event(event_id).await.unwrap(); + let event = store.get_audit_event(tenant_id, event_id).await.unwrap(); assert!(event.is_some(), "Should find the event"); assert_eq!(event.unwrap().id, event_id); println!("✓ Test 10 passed: Get individual event"); + // B1 regression: an event must not be readable from another tenant, even + // by someone who knows its UUID. + let other_tenant = Uuid::new_v4(); + let leaked = store.get_audit_event(other_tenant, event_id).await.unwrap(); + assert!( + leaked.is_none(), + "an audit event must not be readable across tenants (B1)" + ); + println!("✓ Test 10b passed: Audit events are tenant-scoped"); + println!("\n✅ All MongoDB audit logging tests passed!"); } diff --git a/pangolin/pangolin_store/tests/store_integration.rs b/pangolin/pangolin_store/tests/store_integration.rs index ce5ee30..edf66e0 100644 --- a/pangolin/pangolin_store/tests/store_integration.rs +++ b/pangolin/pangolin_store/tests/store_integration.rs @@ -1,5 +1,5 @@ use pangolin_store::{ - tests::{test_asset_update_consistency, test_dashboard_stats_consistency}, + tests::{parity, test_asset_update_consistency, test_dashboard_stats_consistency}, MemoryStore, MongoStore, PostgresStore, SqliteStore, }; use std::env; @@ -10,6 +10,7 @@ async fn test_memory_store_regression() { let store = MemoryStore::new(); test_asset_update_consistency(&store).await; test_dashboard_stats_consistency(&store).await; + parity::run_all(&store).await; } #[tokio::test] @@ -32,6 +33,7 @@ async fn test_sqlite_store_regression() { test_asset_update_consistency(&store).await; test_dashboard_stats_consistency(&store).await; + parity::run_all(&store).await; } #[tokio::test] @@ -50,6 +52,7 @@ async fn test_postgres_store_regression() { test_asset_update_consistency(&store).await; test_dashboard_stats_consistency(&store).await; + parity::run_all(&store).await; } #[tokio::test] @@ -70,4 +73,5 @@ async fn test_mongo_store_regression() { test_asset_update_consistency(&store).await; test_dashboard_stats_consistency(&store).await; + parity::run_all(&store).await; } From 93fdd71d045cdf63326d37c5f88ec73e9fca9015 Mon Sep 17 00:00:00 2001 From: Alex Merced Date: Mon, 10 Aug 2026 10:33:50 -0400 Subject: [PATCH 03/23] fix(deploy,docs): make the quick start work and stop the config drifting Addresses B8-B10 and B43-B46 of roadmap_aug10.md, plus improvement #3. Deployment B8 `docker compose up` could not start the API: the quick-start compose file set no PANGOLIN_JWT_SECRET, and since 0.6.0 the server refuses to start without one. The result was a crash-looping container with no explanation. The `:?` form now fails `docker compose up` itself with a message telling you what to set. B9 Both compose files set PANGOLIN_STORE_TYPE. The server reads PANGOLIN_STORAGE_TYPE, so the variable did nothing - and anyone editing it to `postgres` silently stayed on the in-memory backend and lost their data on every restart. B10 docker-compose.release.yml pinned alexmerced/pangolin-api:0.2.0, four releases behind, and ran scripts/test_release_v0.2.0.py, which does not exist. The "release verification" file verified nothing. Now parameterised by PANGOLIN_VERSION and pointed at a real script. Docs B43 docs/environment-variables.md documented PANGOLIN_HOST, PANGOLIN_PORT and PANGOLIN_STORE_TYPE - none of which any code reads - and omitted 34 that it does. Rewritten from the source, and the second, separately drifted copy under getting-started/ is now a redirect. The fix is not the rewrite; it is scripts/check_env_var_docs.sh, which re-derives the set from the code and fails CI on either kind of drift. A `` marker lets the page still warn readers off names that do not exist. Hygiene B44 A live PyPI API token sat in plaintext in the repo-root .env. It was never tracked by git, but .env is read by docker compose - so a publish credential was being handed to containers with no use for it, and it was one .gitignore edit away from leaking. Removed, with a pointer to the keyring / CI-secret flow PUBLISHING.md already describes. ** The token must be rotated at pypi.org: it existed on disk in plaintext and has to be treated as exposed. ** B45 git rm --cached on ~260 KB of tracked debug output, deleted the two .bak store monoliths (~4k lines of divergent dead query copies that any grep of the store layer would hit), and added the ignore patterns that were missing when those files were first committed. B46 Pruned 18 unused runtime dependencies (all @smui/*, marked, smui-theme) from the UI - zero imports in src/, pure image bloat. Added the missing @vitest/coverage-v8 so `npm run test:coverage` runs, added a test:e2e script for the Playwright specs that had none, and pointed Playwright at 5173 (where vite dev actually serves) with webServer enabled, so the specs connect to something. CI (improvement #3) A `config drift` job validates every compose file, asserts the quick start still fails loudly without a signing secret, rejects any PANGOLIN_* name in a compose file that no code reads, and runs the env-var doc check. All three of B8/B9/B10 are the same failure - config nothing validates - so the job targets the class rather than the instances. Co-Authored-By: Claude Opus 5 --- .github/workflows/ci.yml | 56 + .gitignore | 11 + docker-compose.release.yml | 17 +- docker-compose.yml | 18 +- docs/environment-variables.md | 198 +- docs/getting-started/env_vars.md | 67 +- pangolin/.test_output.txt | 391 ---- pangolin/logs/api_log.txt | 417 ---- pangolin/pangolin_store/src/memory.rs.bak | 1831 ------------------ pangolin/pangolin_store/src/mongo.rs.bak | 2112 --------------------- pangolin/scripts/check_env_var_docs.sh | 78 + pangolin_ui/.gitignore | 9 + pangolin_ui/check_catalogs_list.txt | 971 ---------- pangolin_ui/check_fed_cat.txt | 971 ---------- pangolin_ui/check_final.txt | 1030 ---------- pangolin_ui/check_service_users.txt | 1006 ---------- pangolin_ui/check_service_users_2.txt | 994 ---------- pangolin_ui/package.json | 23 +- pangolin_ui/playwright.config.ts | 13 +- 19 files changed, 385 insertions(+), 9828 deletions(-) delete mode 100644 pangolin/.test_output.txt delete mode 100644 pangolin/logs/api_log.txt delete mode 100644 pangolin/pangolin_store/src/memory.rs.bak delete mode 100644 pangolin/pangolin_store/src/mongo.rs.bak create mode 100755 pangolin/scripts/check_env_var_docs.sh delete mode 100644 pangolin_ui/check_catalogs_list.txt delete mode 100644 pangolin_ui/check_fed_cat.txt delete mode 100644 pangolin_ui/check_final.txt delete mode 100644 pangolin_ui/check_service_users.txt delete mode 100644 pangolin_ui/check_service_users_2.txt diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c66eba9..859fd76 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -206,3 +206,59 @@ jobs: echo "::error::the image runs as root" exit 1 fi + + # Deployment artefacts drift from the code that reads them, and nothing + # noticed. `PANGOLIN_STORE_TYPE` sat in both compose files for several + # releases while the server read `PANGOLIN_STORAGE_TYPE` (B9); the quick-start + # compose file could not start the API at all because it set no signing secret + # (B8); and the release compose file pinned an image four versions old and ran + # a verification script that did not exist (B10). All three are the same + # failure: config that is never validated. + config-drift: + name: config drift + runs-on: ubuntu-latest + defaults: + run: + working-directory: . + steps: + - uses: actions/checkout@v4 + + - name: Every compose file must be valid + env: + PANGOLIN_JWT_SECRET: ci-only-secret-of-adequate-length-0000000 + run: | + for f in docker-compose*.yml; do + echo "validating $f" + docker compose -f "$f" config > /dev/null + done + + - name: The quick start must fail loudly without a signing secret + run: | + # `docker compose config` must *fail* here: the `:?` form on + # PANGOLIN_JWT_SECRET is what turns a crash-looping container into an + # immediate, explicable error. + if docker compose -f docker-compose.yml config > /dev/null 2>&1; then + echo "::error::docker-compose.yml accepted an unset PANGOLIN_JWT_SECRET" + exit 1 + fi + echo "compose correctly refuses to start without a signing secret" + + - name: No compose file may set a variable the server does not read + run: | + # PANGOLIN_STORE_TYPE is the specific name that survived for releases; + # the general check below catches its successors. + if grep -n 'PANGOLIN_STORE_TYPE' docker-compose*.yml deployment_assets 2>/dev/null | grep -v 'not read'; then + echo "::error::PANGOLIN_STORE_TYPE is not read by the server; use PANGOLIN_STORAGE_TYPE" + exit 1 + fi + for name in $(grep -rhoE 'PANGOLIN_[A-Z0-9_]+' docker-compose*.yml | sort -u); do + if ! grep -rq "$name" pangolin/pangolin_api/src pangolin/pangolin_store/src; then + echo "::error::$name appears in a compose file but no code reads it" + exit 1 + fi + done + echo "every PANGOLIN_* name in the compose files is read by the server" + + - name: The environment-variable reference must match the code + working-directory: pangolin + run: ./scripts/check_env_var_docs.sh diff --git a/.gitignore b/.gitignore index fc9a6d0..0f4410d 100644 --- a/.gitignore +++ b/.gitignore @@ -53,3 +53,14 @@ check_log.txt test_results.txt debug_output*.txt :memory: + +# Editor backup copies of source files (B45). +# +# `pangolin_store/src/memory.rs.bak` and `mongo.rs.bak` sat in the tree as ~4k +# lines of divergent, dead query copies - a grep trap that returned plausible +# but stale code for anyone searching the store layer. +*.bak +*.rs.bak + +# Logs live under logs/ directories that were previously tracked. +logs/ diff --git a/docker-compose.release.yml b/docker-compose.release.yml index e2b8cda..05ca9f0 100644 --- a/docker-compose.release.yml +++ b/docker-compose.release.yml @@ -34,12 +34,20 @@ services: " pangolin-api: - image: alexmerced/pangolin-api:0.2.0 + # B10: this pinned 0.2.0 - four releases behind the workspace - so the + # "release verification" compose file verified an image nobody ships. + # Parameterised so a release can be tested by setting PANGOLIN_VERSION, + # with the current version as the default. + image: alexmerced/pangolin-api:${PANGOLIN_VERSION:-0.6.0} ports: - "8080:8080" environment: - RUST_LOG=info - - PANGOLIN_STORE_TYPE=memory + # B9: the server reads PANGOLIN_STORAGE_TYPE; PANGOLIN_STORE_TYPE was + # silently ignored. + - PANGOLIN_STORAGE_TYPE=memory + # B8: the server refuses to start without a JWT secret. + - PANGOLIN_JWT_SECRET=${PANGOLIN_JWT_SECRET:?set it first, e.g. export PANGOLIN_JWT_SECRET=$(openssl rand -base64 48)} - AWS_ACCESS_KEY_ID=minioadmin - AWS_SECRET_ACCESS_KEY=minioadmin - AWS_REGION=us-east-1 @@ -65,7 +73,10 @@ services: - TEST_MODE=${TEST_MODE:-no-auth} depends_on: - pangolin-api - command: sh -c "pip install requests pyiceberg pyarrow && python scripts/test_release_v0.2.0.py" + # B10: this ran scripts/test_release_v0.2.0.py, which does not exist in + # scripts/ - so the verification step failed on a missing file rather than + # verifying anything. + command: sh -c "pip install requests pyiceberg pyarrow && python scripts/integration_test.py" volumes: minio_data_release: diff --git a/docker-compose.yml b/docker-compose.yml index 62c3d12..20e3a48 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -40,7 +40,17 @@ services: - "8080:8080" environment: - RUST_LOG=info - - PANGOLIN_STORE_TYPE=memory + # B8: since 0.6.0 the server refuses to start without a JWT secret, and + # refuses PANGOLIN_NO_AUTH on the default 0.0.0.0 bind - so the documented + # quick start produced a crash-looping container with no explanation. The + # `:?` form fails the `docker compose up` itself with this message, which + # is a far better failure than a restart loop. + - PANGOLIN_JWT_SECRET=${PANGOLIN_JWT_SECRET:?set it first, e.g. export PANGOLIN_JWT_SECRET=$(openssl rand -base64 48)} + # B9: the server reads PANGOLIN_STORAGE_TYPE. This said + # PANGOLIN_STORE_TYPE, which nothing reads - so anyone editing it to + # `postgres` silently kept the in-memory backend and lost their data on + # every restart. + - PANGOLIN_STORAGE_TYPE=memory - AWS_ACCESS_KEY_ID=minioadmin - AWS_SECRET_ACCESS_KEY=minioadmin - AWS_REGION=us-east-1 @@ -57,7 +67,11 @@ services: ports: - "3000:3000" environment: - - VITE_API_URL=http://localhost:8080 + # B31: SvelteKit's dynamic public env requires the PUBLIC_ prefix. The + # client reads PUBLIC_API_URL; this used to pass VITE_API_URL, which + # nothing read, so every deployed build fell back to the *end user's* + # localhost:8080. + - PUBLIC_API_URL=http://localhost:8080 - ORIGIN=http://localhost:3000 depends_on: - pangolin-api diff --git a/docs/environment-variables.md b/docs/environment-variables.md index 60ed622..08486da 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -1,6 +1,13 @@ # Environment Variables Reference -This document provides a comprehensive reference for all environment variables used by Pangolin API. +This document provides a comprehensive reference for all environment variables +used by the Pangolin API. + +> **This page is checked against the code.** `pangolin/scripts/check_env_var_docs.sh` +> re-derives the set of `PANGOLIN_*` variables the server actually reads and +> fails if this file documents one that nothing reads, or omits one that +> something does. It runs in CI. Before that check existed, this page listed +> three variables that did not exist and omitted thirty-four that did. ## Table of Contents @@ -234,34 +241,187 @@ export DATABASE_URL="/var/lib/pangolin/pangolin.db" ## Server Configuration -### `PANGOLIN_HOST` +### `PANGOLIN_BIND_ADDRESS` -**Required:** No -**Type:** String (IP address) -**Default:** `0.0.0.0` +**Required:** No +**Type:** String (IP address) +**Default:** `0.0.0.0` **Description:** IP address to bind the server to. +Note: if `PANGOLIN_NO_AUTH=true`, this **must** be a loopback address. The +server refuses to start otherwise, unconditionally - `PANGOLIN_DEV_MODE` does +not waive the check. + ```bash # Listen on all interfaces -export PANGOLIN_HOST="0.0.0.0" +export PANGOLIN_BIND_ADDRESS="0.0.0.0" -# Listen only on localhost -export PANGOLIN_HOST="127.0.0.1" +# Listen only on localhost (required when PANGOLIN_NO_AUTH=true) +export PANGOLIN_BIND_ADDRESS="127.0.0.1" ``` -### `PANGOLIN_PORT` or `PORT` +### `PORT` -**Required:** No -**Type:** Integer -**Default:** `8080` +**Required:** No +**Type:** Integer +**Default:** `8080` **Description:** Port number for the API server. ```bash -export PANGOLIN_PORT=3000 -# or export PORT=3000 ``` +### `PANGOLIN_MAX_BODY_BYTES` + +**Required:** No +**Type:** Integer (bytes) +**Default:** 10 MiB +**Description:** Largest request body the server will buffer. + +### `PANGOLIN_REQUEST_TIMEOUT_SECS` + +**Required:** No +**Type:** Integer (seconds) +**Default:** `30` +**Description:** Deadline for a request, **including** time spent queued behind +the concurrency limiter. + +### `PANGOLIN_MAX_CONCURRENT_REQUESTS` + +**Required:** No +**Type:** Integer +**Default:** `512` +**Description:** Requests admitted concurrently; the rest queue, bounded by the +request timeout above. + +### `PANGOLIN_SHUTDOWN_GRACE_SECS` + +**Required:** No +**Type:** Integer (seconds) +**Description:** Upper bound on the drain after SIGTERM. Readiness fails +immediately; in-flight requests then have this long to finish before the +process exits regardless. Set it below your orchestrator's termination grace +period. + +### `PANGOLIN_CORS_ALLOWED_ORIGINS` + +**Required:** No +**Type:** Comma-separated list of origins +**Default:** any origin +**Description:** Restricts CORS to the listed origins. + +### `PANGOLIN_WAREHOUSE_CACHE_TTL_SECS` + +**Required:** No +**Type:** Integer (seconds) +**Default:** `5` +**Description:** TTL of the warehouse cache. Entries hold cloud storage +credentials and the cache is node-local, so with more than one replica a +rotated credential can still be vended by peers for up to this long. + +### `PANGOLIN_METRICS_ENABLED` + +**Required:** No +**Type:** Boolean +**Description:** Serve Prometheus metrics at `/metrics`. + +--- + +## Identity and Access + +### `PANGOLIN_JWT_SECRET` + +**Required:** **Yes**, unless `PANGOLIN_DEV_MODE` or `PANGOLIN_NO_AUTH` is set +**Type:** String (32+ characters) +**Description:** Signing key for session JWTs. The server refuses to start +without it, and rejects known-weak values. + +```bash +export PANGOLIN_JWT_SECRET="$(openssl rand -base64 48)" +``` + +### `PANGOLIN_ROOT_USER` / `PANGOLIN_ROOT_PASSWORD` + +**Required:** No +**Type:** String +**Description:** Credentials for the root basic-auth principal. Compared in +constant time. A weak password is refused outside dev mode. + +### `PANGOLIN_SEED_ADMIN`, `PANGOLIN_ADMIN_USER`, `PANGOLIN_ADMIN_PASSWORD` + +**Required:** No +**Type:** Boolean / String / String +**Description:** Seed a first tenant administrator on an empty database. + +### `PANGOLIN_SESSION_TTL_SECS` + +**Required:** No +**Type:** Integer (seconds) +**Description:** Lifetime of an issued session token. + +### `PANGOLIN_DEV_MODE` + +**Required:** No +**Type:** Boolean +**Default:** `false` +**Description:** Relaxes *secret strength* requirements for local development. +It does **not** relax network exposure: the loopback requirement on +`PANGOLIN_NO_AUTH` applies regardless. + +### `PANGOLIN_ALLOW_LEGACY_API_KEYS` + +**Required:** No +**Type:** Boolean +**Default:** `false` +**Description:** Accept API keys minted before the key-id format. Off by +default because a legacy key costs a bcrypt verification per candidate. + +### `PANGOLIN_ALLOW_TOKENS_WITHOUT_JTI` + +**Required:** No +**Type:** Boolean +**Default:** `false` +**Description:** Accept JWTs carrying no `jti`. Such a token can never be +revoked, so this exists only as a migration window. + +--- + +## OAuth / OIDC + +Each provider is enabled by setting its client id and secret. The redirect URI +defaults to `/oauth/callback/`. + +| Provider | Variables | +|-----------|-----------| +| Google | `PANGOLIN_GOOGLE_CLIENT_ID`, `PANGOLIN_GOOGLE_CLIENT_SECRET`, `PANGOLIN_GOOGLE_REDIRECT_URI` | +| GitHub | `PANGOLIN_GITHUB_CLIENT_ID`, `PANGOLIN_GITHUB_CLIENT_SECRET`, `PANGOLIN_GITHUB_REDIRECT_URI` | +| Microsoft | `PANGOLIN_MICROSOFT_CLIENT_ID`, `PANGOLIN_MICROSOFT_CLIENT_SECRET`, `PANGOLIN_MICROSOFT_REDIRECT_URI`, `PANGOLIN_MICROSOFT_TENANT_ID` | +| Okta | `PANGOLIN_OKTA_CLIENT_ID`, `PANGOLIN_OKTA_CLIENT_SECRET`, `PANGOLIN_OKTA_REDIRECT_URI`, `PANGOLIN_OKTA_DOMAIN` | + +### `FRONTEND_URL` + +**Required:** No +**Type:** URL +**Default:** `http://localhost:5173` +**Description:** Where the UI lives. Always an acceptable OAuth landing page. + +### `PANGOLIN_OAUTH_REDIRECT_URIS` + +**Required:** No +**Type:** Comma-separated list of exact URLs +**Description:** Additional URLs an OAuth flow may hand control back to. +Anything not listed (and not `FRONTEND_URL`) is refused. + +### `PANGOLIN_OAUTH_EMAIL_LINK_DOMAINS` + +**Required:** No +**Type:** Comma-separated list of domains +**Default:** empty +**Description:** Domains whose **verified** addresses may adopt a pre-existing +local account. With no allowlist, identity is `(provider, subject)` only and an +email address never links an account - which is what stops someone setting a +matching address on any configured provider and logging in as that user. + --- ## Logging @@ -316,7 +476,7 @@ export AWS_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE" export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" export AWS_REGION="us-east-1" export RUST_LOG=info -export PANGOLIN_PORT=8080 +export PORT=8080 ./pangolin_api ``` @@ -379,8 +539,10 @@ services: ### ❌ Using the wrong variable names ```bash -# WRONG - These are not used -export PANGOLIN_STORE_TYPE=mongo +# WRONG - This one is not read by anything. It appeared in the compose files +# and in this document for several releases, so editing it to `postgres` +# silently left you on the in-memory backend. +export PANGOLIN_STORE_TYPE=mongo # export MONGODB_URI=mongodb://localhost:27017 export MONGODB_DATABASE=pangolin @@ -433,7 +595,7 @@ When multiple variables could configure the same thing: 1. **Storage Backend:** `DATABASE_URL` (only this is used) 2. **MongoDB Database:** `MONGO_DB_NAME` (only this is used) 3. **S3 Endpoint:** `S3_ENDPOINT` or `AWS_ENDPOINT_URL` (both work) -4. **Server Port:** `PANGOLIN_PORT` or `PORT` (PANGOLIN_PORT takes precedence) +4. **Server Port:** `PORT` --- diff --git a/docs/getting-started/env_vars.md b/docs/getting-started/env_vars.md index 32ae9cf..b9ace1a 100644 --- a/docs/getting-started/env_vars.md +++ b/docs/getting-started/env_vars.md @@ -1,63 +1,12 @@ # Environment Variables -Pangolin is configured via environment variables. This guide lists all available options. +**This page has moved to [../environment-variables.md](../environment-variables.md).** -## 🚀 Core API Configuration +Two hand-maintained environment-variable references existed side by side and +both had drifted from the code (B43). Keeping one of them was not a fix - the +problem was that either could drift silently. -| Variable | Description | Default | -|----------|-------------|---------| -| `PORT` | The port the API server will listen on. | `8080` | -| `RUST_LOG` | Log level (`error`, `warn`, `info`, `debug`, `trace`). | `info` | - -## 💾 Metadata Persistence - -Pangolin stores its own metadata (tenants, users, catalogs) in a backend database. - -| Variable | Description | Default | -|----------|-------------|---------| -| `DATABASE_URL` | Connection string for Postgres, MongoDB, or SQLite. | (None) | -| `PANGOLIN_STORAGE_TYPE` | Storage driver if `DATABASE_URL` is missing. (`memory`, `postgres`, `mongo`, `sqlite`). | `memory` | -| `MONGO_DB_NAME` | Database name (when using MongoDB). | `pangolin` | - -> [!NOTE] -> If `DATABASE_URL` is provided, the driver is automatically inferred from the URI scheme (e.g., `postgresql://`, `mongodb://`, `sqlite://`). - -## 🛡️ Authentication & Security - -| Variable | Description | Default | -|----------|-------------|---------| -| `PANGOLIN_NO_AUTH` | **Evaluation Mode**. Auto-provisions a default tenant and admin user. Set to `"true"` to enable. | `false` | -| `PANGOLIN_JWT_SECRET` | Secret key for JWT signing. **MUST** be changed in production. | `default_secret` | -| `PANGOLIN_SEED_ADMIN` | Auto-provision a Tenant Admin even if `NO_AUTH` is false. Set to `"true"`. | `false` | -| `PANGOLIN_ADMIN_USER` | Username for seed admin. | `tenant_admin` | -| `PANGOLIN_ADMIN_PASSWORD` | Password for seed admin. | `password123` | -| `PANGOLIN_ROOT_USER` | Initial Root user for multi-tenant bootstrapping. | `admin` | -| `PANGOLIN_ROOT_PASSWORD` | Password for Root user. | `password` | - -## 🌐 OAuth 2.0 (External Providers) - -Required for enabling "Login with Google/GitHub/Microsoft" in the UI. - -| Variable | Provider | -|----------|----------| -| `OAUTH_GOOGLE_CLIENT_ID` / `_SECRET` | Google | -| `OAUTH_MICROSOFT_CLIENT_ID` / `_SECRET` | Microsoft | -| `OAUTH_GITHUB_CLIENT_ID` / `_SECRET` | GitHub | - -## ☁️ Cloud Provider Features - -When building with cloud features (`--features aws-sts`, etc.), these standard variables are used by the underlying SDKs for the **Signer** logic. - -| Variable | Description | -|----------|-------------| -| `AWS_ACCESS_KEY_ID` | AWS Credentials for STS / Signer. | -| `AWS_SECRET_ACCESS_KEY` | AWS Credentials for STS / Signer. | -| `AWS_REGION` | Default AWS region. | -| `AWS_ENDPOINT_URL` | Override for MinIO or custom S3 backends. | - -## 🚦 Security Checklist - -1. **Disable NO_AUTH**: Ensure `PANGOLIN_NO_AUTH` is unset or `false` in production. -2. **Rotate JWT Secret**: Use a 32+ character random string. -3. **Strong Root Password**: Change the default `admin/password` immediately. -4. **Use DATABASE_URL**: Avoid the `memory` store for any non-trivial use case. +The surviving reference is checked against the source in CI by +`pangolin/scripts/check_env_var_docs.sh`, which fails the build if it documents +a `PANGOLIN_*` variable nothing reads, or omits one something does. This page is +a redirect so existing links keep working. diff --git a/pangolin/.test_output.txt b/pangolin/.test_output.txt deleted file mode 100644 index 643cc43..0000000 --- a/pangolin/.test_output.txt +++ /dev/null @@ -1,391 +0,0 @@ -warning: unused import: `std::collections::HashMap` - --> pangolin_core/src/business_metadata.rs:3:5 - | -3 | use std::collections::HashMap; - | ^^^^^^^^^^^^^^^^^^^^^^^^^ - | - = note: `#[warn(unused_imports)]` (part of `#[warn(unused)]`) on by default - -warning: `pangolin_core` (lib) generated 1 warning (run `cargo fix --lib -p pangolin_core` to apply 1 suggestion) -warning: unused import: `CatalogType` - --> pangolin_store/src/memory.rs:7:14 - | -7 | ...g, CatalogType, N... - | ^^^^^^^^^^^ - | - = note: `#[warn(unused_imports)]` (part of `#[warn(unused)]`) on by default - -warning: unused import: `BusinessMetadata` - --> pangolin_store/src/memory.rs:16:40 - | -16 | ...::{BusinessMetadata, A... - | ^^^^^^^^^^^^^^^^ - -warning: unused import: `PermissionGrant` - --> pangolin_store/src/postgres.rs:9:51 - | -9 | ...n, PermissionGrant, U... - | ^^^^^^^^^^^^^^^ - -warning: unused import: `DateTime` - --> pangolin_store/src/postgres.rs:16:14 - | -16 | ...::{DateTime, U... - | ^^^^^^^^ - -warning: unused imports: `OAuthProvider` and `UserRole as CoreUserRole` - --> pangolin_store/src/mongo.rs:13:33 - | -13 | ...r, UserRole as CoreUserRole, OAuthProvider}; - | ^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^ - -warning: unused import: `PermissionGrant` - --> pangolin_store/src/mongo.rs:14:51 - | -14 | ...n, PermissionGrant, U... - | ^^^^^^^^^^^^^^^ - -warning: unused import: `RequestStatus` - --> pangolin_store/src/mongo.rs:15:55 - | -15 | ...t, RequestStatus}; - | ^^^^^^^^^^^^^ - -warning: unused import: `PermissionGrant` - --> pangolin_store/src/sqlite.rs:5:51 - | -5 | ...n, PermissionGrant, U... - | ^^^^^^^^^^^^^^^ - -warning: unused import: `DateTime` - --> pangolin_store/src/sqlite.rs:12:14 - | -12 | ...::{DateTime, U... - | ^^^^^^^^ - -warning: unused import: `crate::memory::MemoryStore` - --> pangolin_store/src/tests/multi_cloud.rs:1:5 - | -1 | use crate::memory::MemoryStore; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: unused import: `crate::CatalogStore` - --> pangolin_store/src/tests/multi_cloud.rs:2:5 - | -2 | use crate::CatalogStore; - | ^^^^^^^^^^^^^^^^^^^ - -warning: unused imports: `Credentials` and `Signer` - --> pangolin_store/src/tests/multi_cloud.rs:3:21 - | -3 | ...::{Signer, Credentials}; - | ^^^^^^ ^^^^^^^^^^^ - -warning: unused imports: `Tenant`, `VendingStrategy`, and `Warehouse` - --> pangolin_store/src/tests/multi_cloud.rs:4:28 - | -4 | ...::{Warehouse, VendingStrategy, Tenant}; - | ^^^^^^^^^ ^^^^^^^^^^^^^^^ ^^^^^^ - -warning: unused import: `std::collections::HashMap` - --> pangolin_store/src/tests/multi_cloud.rs:5:5 - | -5 | use std::collections::HashMap; - | ^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: unused import: `uuid::Uuid` - --> pangolin_store/src/tests/multi_cloud.rs:6:5 - | -6 | use uuid::Uuid; - | ^^^^^^^^^^ - -warning: unused imports: `AuditAction`, `AuditLogEntry`, `AuditLogFilter`, `AuditResult`, and `ResourceType` - --> pangolin_store/src/tests/audit_tests.rs:1:28 - | - 1 | ...::{AuditAction, AuditLogEntry, AuditLogFilter, AuditResult, ResourceType}; - | ^^^^^^^^^^^ ^^^^^^^^^^^^^ ^^^^^^^^^^^^^^ ^^^^^^^^^^^ ^^^^^^^^^^^^ - | -help: if this is a test module, consider adding a `#[cfg(test)]` to the containing module - --> pangolin_store/src/tests/mod.rs:154:1 - | -154 | pub mod audit_tests; - | ^^^^^^^^^^^^^^^^^^^^ - -warning: unused imports: `CatalogStore` and `MemoryStore` - --> pangolin_store/src/tests/audit_tests.rs:2:13 - | - 2 | ...::{CatalogStore, MemoryStore}; - | ^^^^^^^^^^^^ ^^^^^^^^^^^ - | -help: if this is a test module, consider adding a `#[cfg(test)]` to the containing module - --> pangolin_store/src/tests/mod.rs:154:1 - | -154 | pub mod audit_tests; - | ^^^^^^^^^^^^^^^^^^^^ - -warning: unused import: `uuid::Uuid` - --> pangolin_store/src/tests/audit_tests.rs:3:5 - | - 3 | use uuid::Uuid; - | ^^^^^^^^^^ - | -help: if this is a test module, consider adding a `#[cfg(test)]` to the containing module - --> pangolin_store/src/tests/mod.rs:154:1 - | -154 | pub mod audit_tests; - | ^^^^^^^^^^^^^^^^^^^^ - -warning: unused import: `chrono::Utc` - --> pangolin_store/src/tests/audit_tests.rs:4:5 - | - 4 | use chrono::Utc; - | ^^^^^^^^^^^ - | -help: if this is a test module, consider adding a `#[cfg(test)]` to the containing module - --> pangolin_store/src/tests/mod.rs:154:1 - | -154 | pub mod audit_tests; - | ^^^^^^^^^^^^^^^^^^^^ - -warning: unused imports: `Duration` and `Utc` - --> pangolin_store/src/azure_signer.rs:2:14 - | -2 | ...::{Duration, Utc}; - | ^^^^^^^^ ^^^ - -warning: unused import: `azure_storage_blobs::prelude::*` - --> pangolin_store/src/azure_signer.rs:4:5 - | -4 | use azure_storage_blobs::prelude::*; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: unused imports: `DateTime` and `Utc` - --> pangolin_store/src/gcp_signer.rs:2:14 - | -2 | ...::{DateTime, Utc}; - | ^^^^^^^^ ^^^ - -warning: unused import: `std::sync::Arc` - --> pangolin_store/src/object_store_factory.rs:4:5 - | -4 | use std::sync::Arc; - | ^^^^^^^^^^^^^^ - -warning: unused import: `TimeZone` - --> pangolin_store/src/postgres.rs:16:29 - | -16 | ...c, TimeZone}; - | ^^^^^^^^ - -warning: unused variable: `now` - --> pangolin_store/src/memory.rs:1184:13 - | -1184 | ...et now = ... - | ^^^ help: if this is intentional, prefix it with an underscore: `_now` - | - = note: `#[warn(unused_variables)]` (part of `#[warn(unused)]`) on by default - -warning: unused variable: `client` - --> pangolin_store/src/gcp_signer.rs:52:13 - | -52 | ...et client = ... - | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_client` - -warning: value assigned to `param_count` is never read - --> pangolin_store/src/postgres.rs:1168:17 - | -1168 | ... param_count += 1; - | ^^^^^^^^^^^^^^^^ - | - = help: maybe it is overwritten before being read? - = note: `#[warn(unused_assignments)]` (part of `#[warn(unused)]`) on by default - -warning: value assigned to `param_count` is never read - --> pangolin_store/src/postgres.rs:1312:17 - | -1312 | ... param_count += 1; - | ^^^^^^^^^^^^^^^^ - | - = help: maybe it is overwritten before being read? - -warning: unused variable: `tenant_id` - --> pangolin_store/src/mongo.rs:178:38 - | -178 | ...f, tenant_id: U... - | ^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_tenant_id` - -warning: variable does not need to be mutable - --> pangolin_store/src/mongo.rs:357:13 - | -357 | ...et mut filter = ... - | ----^^^^^^ - | | - | help: remove this `mut` - | - = note: `#[warn(unused_mut)]` (part of `#[warn(unused)]`) on by default - -warning: unused variable: `p` - --> pangolin_store/src/mongo.rs:362:21 - | -362 | ...me(p) = pa... - | ^ help: if this is intentional, prefix it with an underscore: `_p` - -warning: unused variable: `s3_nested` - --> pangolin_store/src/mongo.rs:1662:18 - | -1662 | ...et s3_nested = ... - | ^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_s3_nested` - -warning: field `signer` is never read - --> pangolin_store/src/memory.rs:37:5 - | -22 | pub struct MemoryStore { - | ----------- field in this struct -... -37 | signer: crate::signer... - | ^^^^^^ - | - = note: `MemoryStore` has a derived impl for the trait `Clone`, but this is intentionally ignored during dead code analysis - = note: `#[warn(dead_code)]` (part of `#[warn(unused)]`) on by default - -warning: field `key` is never read - --> pangolin_store/src/signer.rs:41:5 - | -40 | pub struct SignerImpl { - | ---------- field in this struct -41 | key: String, - | ^^^ - | - = note: `SignerImpl` has a derived impl for the trait `Clone`, but this is intentionally ignored during dead code analysis - -warning: methods `revoke_token`, `is_token_revoked`, and `cleanup_expired_tokens` are never used - --> pangolin_store/src/postgres.rs:1809:14 - | -1750 | impl PostgresStore { - | ------------------ methods in this implementation -... -1809 | async fn revoke_token(&self, token... - | ^^^^^^^^^^^^ -... -1821 | async fn is_token_revoked(&self, t... - | ^^^^^^^^^^^^^^^^ -... -1831 | async fn cleanup_expired_tokens(&s... - | ^^^^^^^^^^^^^^^^^^^^^^ - -warning: field `client` is never read - --> pangolin_store/src/mongo.rs:27:5 - | -26 | pub struct MongoStore { - | ---------- field in this struct -27 | client: Client, - | ^^^^^^ - | - = note: `MongoStore` has a derived impl for the trait `Clone`, but this is intentionally ignored during dead code analysis - -warning: methods `catalogs`, `branches`, `tags`, `commits`, and `audit_logs` are never used - --> pangolin_store/src/mongo.rs:63:8 - | -31 | impl MongoStore { - | --------------- methods in this implementation -... -63 | fn catalogs(&sel... - | ^^^^^^^^ -... -75 | fn branches(&sel... - | ^^^^^^^^ -... -79 | fn tags(&self) -... - | ^^^^ -... -83 | fn commits(&self... - | ^^^^^^^ -... -87 | fn audit_logs(&s... - | ^^^^^^^^^^ - -warning: methods `revoke_token`, `is_token_revoked`, and `cleanup_expired_tokens` are never used - --> pangolin_store/src/sqlite.rs:2003:14 - | -1941 | impl SqliteStore { - | ---------------- methods in this implementation -... -2003 | async fn revoke_token(&self, token... - | ^^^^^^^^^^^^ -... -2018 | async fn is_token_revoked(&self, t... - | ^^^^^^^^^^^^^^^^ -... -2029 | async fn cleanup_expired_tokens(&s... - | ^^^^^^^^^^^^^^^^^^^^^^ - -warning: function `test_azure_path_parsing` is never used - --> pangolin_store/src/tests/multi_cloud.rs:129:4 - | -129 | fn test_azure_path_parsing() { - | ^^^^^^^^^^^^^^^^^^^^^^^ - -warning: fields `account_name` and `account_key` are never read - --> pangolin_store/src/azure_signer.rs:7:5 - | -6 | pub struct AzureSigner { - | ----------- fields in this struct -7 | account_name: String, - | ^^^^^^^^^^^^ -8 | account_key: String, - | ^^^^^^^^^^^ - -warning: `pangolin_store` (lib) generated 40 warnings (run `cargo fix --lib -p pangolin_store` to apply 29 suggestions) -warning: unused import: `CatalogStore` - --> pangolin_store/tests/store_integration.rs:2:58 - | -2 | ...e, CatalogStore, - | ^^^^^^^^^^^^ - | - = note: `#[warn(unused_imports)]` (part of `#[warn(unused)]`) on by default - -warning: `pangolin_store` (test "store_integration") generated 1 warning (run `cargo fix --test "store_integration" -p pangolin_store` to apply 1 suggestion) - Finished `test` profile [unoptimized + debuginfo] target(s) in 0.32s - Running tests/store_integration.rs (target/debug/deps/store_integration-9aac1c19d52cf927) - -running 4 tests -test test_memory_store_regression ... ok -test test_sqlite_store_regression ... FAILED -test test_postgres_store_regression ... FAILED -test test_mongo_store_regression ... FAILED - -failures: - ----- test_sqlite_store_regression stdout ---- - -thread 'test_sqlite_store_regression' (314071) panicked at /home/alexmerced/development/personal/Personal/2026/pangolin/pangolin/pangolin_store/src/tests/mod.rs:51:90: -Failed to create asset: error returned from database: (code: 1) no such table: assets - -Caused by: - (code: 1) no such table: assets -note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace - ----- test_postgres_store_regression stdout ---- - -thread 'test_postgres_store_regression' (314070) panicked at /home/alexmerced/development/personal/Personal/2026/pangolin/pangolin/pangolin_store/src/tests/mod.rs:51:90: -Failed to create asset: error returned from database: insert or update on table "assets" violates foreign key constraint "assets_tenant_id_fkey" - -Caused by: - insert or update on table "assets" violates foreign key constraint "assets_tenant_id_fkey" - ----- test_mongo_store_regression stdout ---- - -thread 'test_mongo_store_regression' (314069) panicked at /home/alexmerced/development/personal/Personal/2026/pangolin/pangolin/pangolin_store/src/tests/mod.rs:56:5: -assertion `left == right` failed: Initial metadata location mismatch - left: None - right: Some("s3://bucket/path/v1.json") - - -failures: - test_mongo_store_regression - test_postgres_store_regression - test_sqlite_store_regression - -test result: FAILED. 1 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.87s - -error: test failed, to rerun pass `-p pangolin_store --test store_integration` diff --git a/pangolin/logs/api_log.txt b/pangolin/logs/api_log.txt deleted file mode 100644 index 5e1615a..0000000 --- a/pangolin/logs/api_log.txt +++ /dev/null @@ -1,417 +0,0 @@ -warning: unused import: `std::collections::HashMap` - --> pangolin_store/src/postgres.rs:10:5 - | -10 | use std::collections::HashMap; - | ^^^^^^^^^^^^^^^^^^^^^^^^^ - | - = note: `#[warn(unused_imports)]` (part of `#[warn(unused)]`) on by default - -warning: unused variable: `target` - --> pangolin_store/src/postgres.rs:637:13 - | -637 | let target = self.get_branch(tenant_id, catalog_name, target_branch.clone()).await? - | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_target` - | - = note: `#[warn(unused_variables)]` (part of `#[warn(unused)]`) on by default - -warning: unused variable: `location` - --> pangolin_store/src/postgres.rs:697:31 - | -697 | async fn read_file(&self, location: &str) -> Result> { - | ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_location` - -warning: unused variable: `bytes` - --> pangolin_store/src/postgres.rs:703:48 - | -703 | async fn write_file(&self, location: &str, bytes: Vec) -> Result<()> { - | ^^^^^ help: if this is intentional, prefix it with an underscore: `_bytes` - -warning: unused variable: `location` - --> pangolin_store/src/postgres.rs:703:32 - | -703 | async fn write_file(&self, location: &str, bytes: Vec) -> Result<()> { - | ^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_location` - -warning: unused variable: `branch` - --> pangolin_store/src/postgres.rs:820:83 - | -820 | ...d: Uuid, catalog_name: &str, branch: Option, namespace: Vec, table: String, expected_location: Option, new_loc... - | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_branch` - -warning: unused variable: `tenant_id` - --> pangolin_store/src/mongo.rs:121:38 - | -121 | async fn create_warehouse(&self, tenant_id: Uuid, warehouse: Warehouse) -> Result<()> { - | ^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_tenant_id` - -warning: variable does not need to be mutable - --> pangolin_store/src/mongo.rs:284:13 - | -284 | let mut filter = doc! { - | ----^^^^^^ - | | - | help: remove this `mut` - | - = note: `#[warn(unused_mut)]` (part of `#[warn(unused)]`) on by default - -warning: unused variable: `p` - --> pangolin_store/src/mongo.rs:289:21 - | -289 | if let Some(p) = parent { - | ^ help: if this is intentional, prefix it with an underscore: `_p` - -warning: unused variable: `branch` - --> pangolin_store/src/mongo.rs:364:68 - | -364 | ...d: Uuid, catalog_name: &str, branch: Option, namespace: Vec, name: String) -> Result> { - | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_branch` - -warning: unused variable: `branch` - --> pangolin_store/src/mongo.rs:401:70 - | -401 | async fn list_assets(&self, tenant_id: Uuid, catalog_name: &str, branch: Option, namespace: Vec) -> Result> { - | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_branch` - -warning: unused variable: `branch` - --> pangolin_store/src/mongo.rs:432:71 - | -432 | ...d: Uuid, catalog_name: &str, branch: Option, namespace: Vec, name: String) -> Result<()> { - | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_branch` - -warning: unused variable: `branch` - --> pangolin_store/src/mongo.rs:443:71 - | -443 | ...d: Uuid, catalog_name: &str, branch: Option, source_namespace: Vec, source_name: String, dest_namespace: Vec, ... - | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_branch` - -warning: unused variable: `branch` - --> pangolin_store/src/mongo.rs:714:83 - | -714 | ...d: Uuid, catalog_name: &str, branch: Option, namespace: Vec, table: String, expected_location: Option, new_loc... - | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_branch` - -warning: field `signer` is never read - --> pangolin_store/src/memory.rs:32:5 - | -17 | pub struct MemoryStore { - | ----------- field in this struct -... -32 | signer: crate::signer::SignerImpl, - | ^^^^^^ - | - = note: `MemoryStore` has a derived impl for the trait `Clone`, but this is intentionally ignored during dead code analysis - = note: `#[warn(dead_code)]` (part of `#[warn(unused)]`) on by default - -warning: field `key` is never read - --> pangolin_store/src/signer.rs:25:5 - | -24 | pub struct SignerImpl { - | ---------- field in this struct -25 | key: String, - | ^^^ - | - = note: `SignerImpl` has a derived impl for the trait `Clone`, but this is intentionally ignored during dead code analysis - -warning: field `client` is never read - --> pangolin_store/src/mongo.rs:18:5 - | -17 | pub struct MongoStore { - | ---------- field in this struct -18 | client: Client, - | ^^^^^^ - | - = note: `MongoStore` has a derived impl for the trait `Clone`, but this is intentionally ignored during dead code analysis - -warning: multiple methods are never used - --> pangolin_store/src/mongo.rs:38:8 - | -22 | impl MongoStore { - | --------------- methods in this implementation -... -38 | fn catalogs(&self) -> Collection { - | ^^^^^^^^ -... -42 | fn namespaces(&self) -> Collection { - | ^^^^^^^^^^ -... -46 | fn assets(&self) -> Collection { - | ^^^^^^ -... -50 | fn branches(&self) -> Collection { - | ^^^^^^^^ -... -54 | fn tags(&self) -> Collection { - | ^^^^ -... -58 | fn commits(&self) -> Collection { - | ^^^^^^^ -... -62 | fn audit_logs(&self) -> Collection { - | ^^^^^^^^^^ - -warning: `pangolin_store` (lib) generated 18 warnings (run `cargo fix --lib -p pangolin_store` to apply 14 suggestions) -warning: unused import: `pangolin_store::memory::MemoryStore` - --> pangolin_api/src/lib.rs:4:5 - | -4 | use pangolin_store::memory::MemoryStore; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - | - = note: `#[warn(unused_imports)]` (part of `#[warn(unused)]`) on by default - -warning: unused import: `Any` - --> pangolin_api/src/lib.rs:6:35 - | -6 | use tower_http::cors::{CorsLayer, Any}; - | ^^^ - -warning: unused imports: `Request` and `body::Body` - --> pangolin_api/src/iceberg_handlers.rs:5:43 - | -5 | http::{StatusCode, HeaderMap, Method, Request}, - | ^^^^^^^ -6 | body::Body, - | ^^^^^^^^^^ - -warning: unused import: `pangolin_store::memory::MemoryStore` - --> pangolin_api/src/iceberg_handlers.rs:12:5 - | -12 | use pangolin_store::memory::MemoryStore; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: unused import: `UserRole` - --> pangolin_api/src/iceberg_handlers.rs:1021:27 - | -1021 | use pangolin_core::user::{UserRole, UserSession}; - | ^^^^^^^^ - -warning: unused import: `std::sync::Arc` - --> pangolin_api/src/tenant_handlers.rs:8:5 - | -8 | use std::sync::Arc; - | ^^^^^^^^^^^^^^ - -warning: unused import: `std::collections::HashMap` - --> pangolin_api/src/tenant_handlers.rs:9:5 - | -9 | use std::collections::HashMap; - | ^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: unused import: `std::sync::Arc` - --> pangolin_api/src/warehouse_handlers.rs:8:5 - | -8 | use std::sync::Arc; - | ^^^^^^^^^^^^^^ - -warning: unused import: `std::sync::Arc` - --> pangolin_api/src/asset_handlers.rs:8:5 - | -8 | use std::sync::Arc; - | ^^^^^^^^^^^^^^ - -warning: unused import: `HeaderMap` - --> pangolin_api/src/auth.rs:3:24 - | -3 | http::{StatusCode, HeaderMap}, - | ^^^^^^^^^ - -warning: unused import: `std::sync::Arc` - --> pangolin_api/src/signing_handlers.rs:8:5 - | -8 | use std::sync::Arc; - | ^^^^^^^^^^^^^^ - -warning: unused import: `Json` - --> pangolin_api/src/oauth_handlers.rs:5:5 - | -5 | Json, - | ^^^^ - -warning: unused import: `Serialize` - --> pangolin_api/src/oauth_handlers.rs:7:26 - | -7 | use serde::{Deserialize, Serialize}; - | ^^^^^^^^^ - -warning: unused import: `std::sync::Arc` - --> pangolin_api/src/business_metadata_handlers.rs:8:5 - | -8 | use std::sync::Arc; - | ^^^^^^^^^^^^^^ - -warning: unused imports: `MergeConflict` and `MergeOperation` - --> pangolin_api/src/merge_handlers.rs:7:48 - | -7 | use pangolin_core::model::{ConflictResolution, MergeConflict, MergeOperation, ResolutionStrategy}; - | ^^^^^^^^^^^^^ ^^^^^^^^^^^^^^ - -warning: unused import: `Serialize` - --> pangolin_api/src/permission_handlers.rs:8:26 - | -8 | use serde::{Deserialize, Serialize}; - | ^^^^^^^^^ - -warning: unused import: `Serialize` - --> pangolin_api/src/service_user_handlers.rs:10:26 - | -10 | use serde::{Deserialize, Serialize}; - | ^^^^^^^^^ - -warning: variable does not need to be mutable - --> pangolin_api/src/auth.rs:92:21 - | -92 | let mut validation = Validation::new(Algorithm::HS256); - | ----^^^^^^^^^^ - | | - | help: remove this `mut` - | - = note: `#[warn(unused_mut)]` (part of `#[warn(unused)]`) on by default - -warning: variable does not need to be mutable - --> pangolin_api/src/oauth_handlers.rs:110:14 - | -110 | Some(mut u) => { - | ----^ - | | - | help: remove this `mut` - -warning: variable does not need to be mutable - --> pangolin_api/src/permission_handlers.rs:100:10 - | -100 | Json(mut role): Json, - | ----^^^^ - | | - | help: remove this `mut` - -warning: unused import: `pangolin_store::CatalogStore` - --> pangolin_api/src/business_metadata_handlers.rs:9:5 - | -9 | use pangolin_store::CatalogStore; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: unused import: `pangolin_store::CatalogStore` - --> pangolin_api/src/tenant_handlers.rs:10:5 - | -10 | use pangolin_store::CatalogStore; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: unused import: `pangolin_store::CatalogStore` - --> pangolin_api/src/warehouse_handlers.rs:9:5 - | -9 | use pangolin_store::CatalogStore; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: unused import: `pangolin_store::CatalogStore` - --> pangolin_api/src/asset_handlers.rs:9:5 - | -9 | use pangolin_store::CatalogStore; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: unused import: `pangolin_store::CatalogStore` - --> pangolin_api/src/signing_handlers.rs:9:5 - | -9 | use pangolin_store::CatalogStore; - | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -warning: unused variable: `prefix` - --> pangolin_api/src/iceberg_handlers.rs:286:5 - | -286 | prefix: Option>, - | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_prefix` - | - = note: `#[warn(unused_variables)]` (part of `#[warn(unused)]`) on by default - -warning: unused variable: `timestamp` - --> pangolin_api/src/iceberg_handlers.rs:656:24 - | -656 | if let Ok(timestamp) = timestamp_str.parse::() { - | ^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_timestamp` - -warning: unused variable: `session` - --> pangolin_api/src/iceberg_handlers.rs:854:15 - | -854 | Extension(session): Extension, - | ^^^^^^^ help: if this is intentional, prefix it with an underscore: `_session` - -warning: unused variable: `session` - --> pangolin_api/src/iceberg_handlers.rs:982:15 - | -982 | Extension(session): Extension, - | ^^^^^^^ help: if this is intentional, prefix it with an underscore: `_session` - -warning: unused variable: `params` - --> pangolin_api/src/pangolin_handlers.rs:66:11 - | -66 | Query(params): Query, - | ^^^^^^ help: if this is intentional, prefix it with an underscore: `_params` - -warning: unused variable: `tenant_uuid` - --> pangolin_api/src/token_handlers.rs:34:9 - | -34 | let tenant_uuid = match Uuid::parse_str(&payload.tenant_id) { - | ^^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_tenant_uuid` - -warning: fields `identifier` and `requirements` are never read - --> pangolin_api/src/iceberg_handlers.rs:175:5 - | -174 | pub struct CommitTableRequest { - | ------------------ fields in this struct -175 | identifier: Option, - | ^^^^^^^^^^ -176 | requirements: Vec, - | ^^^^^^^^^^^^ - | - = note: `#[warn(dead_code)]` (part of `#[warn(unused)]`) on by default - -warning: field `removals` is never read - --> pangolin_api/src/iceberg_handlers.rs:968:5 - | -967 | pub struct UpdateNamespacePropertiesRequest { - | -------------------------------- field in this struct -968 | removals: Option>, - | ^^^^^^^^ - -warning: fields `name` and `catalog` are never read - --> pangolin_api/src/pangolin_handlers.rs:30:5 - | -29 | pub struct ListBranchParams { - | ---------------- fields in this struct -30 | name: Option, - | ^^^^ -31 | catalog: Option, - | ^^^^^^^ - -warning: `pangolin_api` (lib) generated 34 warnings (run `cargo fix --lib -p pangolin_api` to apply 26 suggestions) -warning: unused import: `std::env` - --> pangolin_api/src/main.rs:3:5 - | -3 | use std::env; - | ^^^^^^^^ - | - = note: `#[warn(unused_imports)]` (part of `#[warn(unused)]`) on by default - -warning: unused imports: `Any` and `CorsLayer` - --> pangolin_api/src/main.rs:4:24 - | -4 | use tower_http::cors::{CorsLayer, Any}; - | ^^^^^^^^^ ^^^ - -warning: unused imports: `HeaderValue` and `Method` - --> pangolin_api/src/main.rs:5:18 - | -5 | use axum::http::{HeaderValue, Method}; - | ^^^^^^^^^^^ ^^^^^^ - -warning: unused variable: `storage_type` - --> pangolin_api/src/main.rs:17:9 - | -17 | let storage_type = std::env::var("PANGOLIN_STORAGE_TYPE").unwrap_or_else(|_| "memory".to_string()); - | ^^^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_storage_type` - | - = note: `#[warn(unused_variables)]` (part of `#[warn(unused)]`) on by default - -warning: `pangolin_api` (bin "pangolin_api") generated 4 warnings (run `cargo fix --bin "pangolin_api" -p pangolin_api` to apply 4 suggestions) - Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.25s - Running `target/debug/pangolin_api` -2025-12-14T22:11:31.649619Z  INFO pangolin_api: Using Memory Storage -2025-12-14T22:11:31.650363Z  INFO pangolin_api: Created default tenant for testing: 00000000-0000-0000-0000-000000000000 -2025-12-14T22:11:31.653384Z  INFO pangolin_api: listening on 0.0.0.0:8080 diff --git a/pangolin/pangolin_store/src/memory.rs.bak b/pangolin/pangolin_store/src/memory.rs.bak deleted file mode 100644 index 8ad28a6..0000000 --- a/pangolin/pangolin_store/src/memory.rs.bak +++ /dev/null @@ -1,1831 +0,0 @@ -use crate::CatalogStore; -use crate::signer::{Signer, Credentials}; -use async_trait::async_trait; -use dashmap::DashMap; -use chrono::Utc; -use pangolin_core::model::{ - Catalog, CatalogType, Namespace, Warehouse, Asset, Commit, Branch, Tag, BranchType, Tenant, - VendingStrategy, SystemSettings, SyncStats -}; -use pangolin_core::user::User; -use pangolin_core::permission::{Role, UserRole, Permission}; -use pangolin_core::audit::AuditLogEntry; -use uuid::Uuid; -use anyhow::Result; -use std::sync::Arc; -use pangolin_core::business_metadata::{BusinessMetadata, AccessRequest}; - -use tracing; - - -#[derive(Clone)] -pub struct MemoryStore { - tenants: Arc>, - warehouses: Arc>, // Key: (TenantId, WarehouseName) - catalogs: Arc>, // Key: (TenantId, CatalogName) - namespaces: Arc>, // Key: (TenantId, CatalogName, NamespaceString) - // Key: (TenantId, CatalogName, BranchName, NamespaceString, AssetName) - assets: Arc>, - branches: Arc>, // Key: (TenantId, CatalogName, BranchName) - tags: Arc>, // Key: (TenantId, CatalogName, TagName) - commits: Arc>, // Key: (TenantId, CommitId) - files: Arc>>, // Key: Location - audit_events: Arc>>, // Changed to DashMap for consistency, key tenant_id - // New fields - users: Arc>, - roles: Arc>, - signer: crate::signer::SignerImpl, - user_roles: Arc>, - permissions: Arc>, - business_metadata: Arc>, - access_requests: Arc>, - service_users: Arc>, - merge_operations: Arc>, - merge_conflicts: Arc>, - // Optimization: Direct lookup for assets by ID - // Key: AssetID, Value: (CatalogName, Namespace, Branch, AssetName) - assets_by_id: Arc, Option, String)>>, - // Token revocation - revoked_tokens: Arc>, - // Active tokens (for listing - in real DB this would be querying sessions/tokens table) - // We only store TokenInfo here. The actual validation is stateless JWT + Revocation Check. - // But to "List Tokens", we need to store them. - active_tokens: Arc>, - // System Settings: Tenant -> Settings - system_settings: Arc>, - // Federated Stats: (TenantId, CatalogName) -> SyncStats - federated_stats: Arc>, - // Performance optimizations - object_store_cache: crate::ObjectStoreCache, - metadata_cache: crate::MetadataCache, -} - -impl MemoryStore { - pub fn new() -> Self { - Self { - tenants: Arc::new(DashMap::new()), - warehouses: Arc::new(DashMap::new()), - catalogs: Arc::new(DashMap::new()), - namespaces: Arc::new(DashMap::new()), - assets: Arc::new(DashMap::new()), - branches: Arc::new(DashMap::new()), - tags: Arc::new(DashMap::new()), - commits: Arc::new(DashMap::new()), - files: Arc::new(DashMap::new()), - audit_events: Arc::new(DashMap::new()), - users: Arc::new(DashMap::new()), - roles: Arc::new(DashMap::new()), - signer: crate::signer::SignerImpl::new("memory_key".to_string()), - user_roles: Arc::new(DashMap::new()), - permissions: Arc::new(DashMap::new()), - business_metadata: Arc::new(DashMap::new()), - access_requests: Arc::new(DashMap::new()), - service_users: Arc::new(DashMap::new()), - merge_operations: Arc::new(DashMap::new()), - merge_conflicts: Arc::new(DashMap::new()), - assets_by_id: Arc::new(DashMap::new()), - revoked_tokens: Arc::new(DashMap::new()), - active_tokens: Arc::new(DashMap::new()), - system_settings: Arc::new(DashMap::new()), - federated_stats: Arc::new(DashMap::new()), - object_store_cache: crate::ObjectStoreCache::new(), - metadata_cache: crate::MetadataCache::default(), - } - } -} - - -#[async_trait] -impl CatalogStore for MemoryStore { - async fn create_tenant(&self, tenant: Tenant) -> Result<()> { - self.tenants.insert(tenant.id, tenant); - Ok(()) - } - - async fn get_tenant(&self, tenant_id: Uuid) -> Result> { - if let Some(t) = self.tenants.get(&tenant_id) { - Ok(Some(t.value().clone())) - } else { - Ok(None) - } - } - - async fn list_tenants(&self) -> Result> { - let tenants = self.tenants.iter().map(|t| t.value().clone()).collect(); - Ok(tenants) - } - - async fn update_tenant(&self, tenant_id: Uuid, updates: pangolin_core::model::TenantUpdate) -> Result { - if let Some(mut tenant) = self.tenants.get_mut(&tenant_id) { - if let Some(name) = updates.name { - tenant.name = name; - } - if let Some(properties) = updates.properties { - tenant.properties.extend(properties); - } - Ok(tenant.clone()) - } else { - Err(anyhow::anyhow!("Tenant not found")) - } - } - - async fn delete_tenant(&self, tenant_id: Uuid) -> Result<()> { - if self.tenants.remove(&tenant_id).is_some() { - // TODO: Cascade delete warehouses and catalogs - Ok(()) - } else { - Err(anyhow::anyhow!("Tenant not found")) - } - } - - async fn create_warehouse(&self, tenant_id: Uuid, warehouse: Warehouse) -> Result<()> { - let key = (tenant_id, warehouse.name.clone()); - self.warehouses.insert(key, warehouse); - Ok(()) - } - - async fn get_warehouse(&self, tenant_id: Uuid, name: String) -> Result> { - let key = (tenant_id, name); - if let Some(w) = self.warehouses.get(&key) { - Ok(Some(w.value().clone())) - } else { - Ok(None) - } - } - - async fn list_warehouses(&self, tenant_id: Uuid) -> Result> { - let warehouses = self.warehouses.iter() - .filter(|r| r.key().0 == tenant_id) - .map(|r| r.value().clone()) - .collect(); - Ok(warehouses) - } - - async fn update_warehouse(&self, tenant_id: Uuid, name: String, updates: pangolin_core::model::WarehouseUpdate) -> Result { - let key = (tenant_id, name.clone()); - if let Some(mut warehouse) = self.warehouses.get_mut(&key) { - if let Some(new_name) = updates.name { - // If name is changing, we need to remove old key and insert with new key - let mut w = warehouse.clone(); - w.name = new_name.clone(); - drop(warehouse); // Release the mutable reference - self.warehouses.remove(&key); - let new_key = (tenant_id, new_name); - self.warehouses.insert(new_key, w.clone()); - return Ok(w); - } - if let Some(config) = updates.storage_config { - warehouse.storage_config.extend(config); - } - if let Some(use_sts) = updates.use_sts { - warehouse.use_sts = use_sts; - } - Ok(warehouse.clone()) - } else { - Err(anyhow::anyhow!("Warehouse '{}' not found", name)) - } - } - - async fn delete_warehouse(&self, tenant_id: Uuid, name: String) -> Result<()> { - let key = (tenant_id, name.clone()); - - if self.warehouses.remove(&key).is_some() { - Ok(()) - } else { - Err(anyhow::anyhow!("Warehouse '{}' not found", name)) - } - } - - async fn create_catalog(&self, tenant_id: Uuid, catalog: Catalog) -> Result<()> { - let key = (tenant_id, catalog.name.clone()); - self.catalogs.insert(key, catalog); - Ok(()) - } - - async fn get_catalog(&self, tenant_id: Uuid, name: String) -> Result> { - let key = (tenant_id, name); - if let Some(c) = self.catalogs.get(&key) { - Ok(Some(c.value().clone())) - } else { - Ok(None) - } - } - - async fn list_catalogs(&self, tenant_id: Uuid) -> Result> { - let mut catalogs = Vec::new(); - for entry in self.catalogs.iter() { - let (tid, _) = entry.key(); - if *tid == tenant_id { - catalogs.push(entry.value().clone()); - } - } - Ok(catalogs) - } - - async fn update_catalog(&self, tenant_id: Uuid, name: String, updates: pangolin_core::model::CatalogUpdate) -> Result { - let key = (tenant_id, name.clone()); - if let Some(mut catalog) = self.catalogs.get_mut(&key) { - if let Some(warehouse_name) = updates.warehouse_name { - catalog.warehouse_name = Some(warehouse_name); - } - if let Some(storage_location) = updates.storage_location { - catalog.storage_location = Some(storage_location); - } - if let Some(properties) = updates.properties { - catalog.properties.extend(properties); - } - Ok(catalog.clone()) - } else { - Err(anyhow::anyhow!("Catalog '{}' not found", name)) - } - } - - async fn delete_catalog(&self, tenant_id: Uuid, name: String) -> Result<()> { - let key = (tenant_id, name.clone()); - if self.catalogs.remove(&key).is_some() { - // Cascade delete: Remove all associated resources - // Note: In a real database, this would be handled by foreign keys. - // In MemoryStore, we must manually iterate and remove. - - // Remove Namespaces - self.namespaces.retain(|k, _| !(k.0 == tenant_id && k.1 == name)); - - // Remove Assets - self.assets.retain(|k, _| !(k.0 == tenant_id && k.1 == name)); - - // Remove Branches - self.branches.retain(|k, _| !(k.0 == tenant_id && k.1 == name)); - - // Remove Tags - self.tags.retain(|k, _| !(k.0 == tenant_id && k.1 == name)); - - // Clean up assets_by_id index - // This is expensive O(N) since we have to scan the whole index - // But deletion is rare. - self.assets_by_id.retain(|_, v| v.0 != name); - - Ok(()) - } else { - Err(anyhow::anyhow!("Catalog not found")) - } - } - - async fn create_namespace(&self, tenant_id: Uuid, catalog_name: &str, namespace: Namespace) -> Result<()> { - let key = (tenant_id, catalog_name.to_string(), namespace.to_string()); - self.namespaces.insert(key, namespace); - Ok(()) - } - - async fn list_namespaces(&self, tenant_id: Uuid, catalog_name: &str, parent: Option) -> Result> { - let parent_prefix = parent.unwrap_or_default(); - tracing::info!("DEBUG_MEM: list_namespaces tid={} cat={} parent='{}'", tenant_id, catalog_name, parent_prefix); - let mut namespaces = Vec::new(); - for entry in self.namespaces.iter() { - let (tid, cat, ns_str) = entry.key(); - tracing::info!("DEBUG_MEM: Checking entry tid={} cat={} ns={}", tid, cat, ns_str); - if *tid == tenant_id && cat == catalog_name && (parent_prefix.is_empty() || ns_str.starts_with(&parent_prefix)) { - namespaces.push(entry.value().clone()); - } - } - tracing::info!("DEBUG_MEM: Found {} namespaces", namespaces.len()); - Ok(namespaces) - } - - async fn get_namespace(&self, tenant_id: Uuid, catalog_name: &str, namespace: Vec) -> Result> { - let key = (tenant_id, catalog_name.to_string(), namespace.join(".")); - if let Some(n) = self.namespaces.get(&key) { - Ok(Some(n.value().clone())) - } else { - Ok(None) - } - } - - async fn delete_namespace(&self, tenant_id: Uuid, catalog_name: &str, namespace: Vec) -> Result<()> { - let ns_str = namespace.join("\x1F"); - let key = (tenant_id, catalog_name.to_string(), ns_str); - if self.namespaces.remove(&key).is_some() { - Ok(()) - } else { - Err(anyhow::anyhow!("Namespace not found")) - } - } - - async fn update_namespace_properties(&self, tenant_id: Uuid, catalog_name: &str, namespace: Vec, properties: std::collections::HashMap) -> Result<()> { - let ns_str = namespace.join("\x1F"); - let key = (tenant_id, catalog_name.to_string(), ns_str); - - if let Some(mut ns) = self.namespaces.get_mut(&key) { - ns.properties.extend(properties); - Ok(()) - } else { - Err(anyhow::anyhow!("Namespace not found")) - } - } - - async fn create_asset(&self, tenant_id: Uuid, catalog_name: &str, branch: Option, namespace: Vec, asset: Asset) -> Result<()> { - let branch_name = branch.unwrap_or_else(|| "main".to_string()); - - // 1. Insert Asset - let asset_full_name = format!("{}.{}", namespace.join("."), asset.name); - let key = (tenant_id, catalog_name.to_string(), branch_name.clone(), namespace.join("\x1F"), asset.name.clone()); - self.assets.insert(key, asset.clone()); - - // 2. Update optimized lookups - self.assets_by_id.insert(asset.id, (catalog_name.to_string(), namespace.clone(), Some(branch_name.clone()), asset.name.clone())); - - // 2. Ensure Branch Exists and Update Asset List - let mut branch_obj = self.get_branch(tenant_id, catalog_name, branch_name.clone()).await? - .unwrap_or_else(|| { - Branch { - name: branch_name.clone(), - head_commit_id: None, - branch_type: BranchType::Experimental, - assets: vec![], - } - }); - - if !branch_obj.assets.contains(&asset_full_name) { - branch_obj.assets.push(asset_full_name); - self.create_branch(tenant_id, catalog_name, branch_obj).await?; - } - - Ok(()) - } - - async fn get_asset_by_id(&self, tenant_id: Uuid, asset_id: Uuid) -> Result)>> { - if let Some(entry) = self.assets_by_id.get(&asset_id) { - let (catalog_name, namespace, branch, name) = entry.value().clone(); - // Verify tenant ownership (implicit via proper key lookup) purely for safety - let branch_name = branch.unwrap_or_else(|| "main".to_string()); - let key = (tenant_id, catalog_name.clone(), branch_name, namespace.join("\x1F"), name); - - if let Some(asset) = self.assets.get(&key) { - return Ok(Some((asset.value().clone(), catalog_name, namespace))); - } - } - Ok(None) - } - - async fn get_asset(&self, tenant_id: Uuid, catalog_name: &str, branch: Option, namespace: Vec, name: String) -> Result> { - let branch_name = branch.unwrap_or_else(|| "main".to_string()); - let key = (tenant_id, catalog_name.to_string(), branch_name, namespace.join("\x1F"), name); - if let Some(a) = self.assets.get(&key) { - Ok(Some(a.value().clone())) - } else { - Ok(None) - } - } - - async fn list_assets(&self, tenant_id: Uuid, catalog_name: &str, branch: Option, namespace: Vec) -> Result> { - let branch_name = branch.unwrap_or_else(|| "main".to_string()); - let ns_str = namespace.join("\x1F"); - let mut assets = Vec::new(); - for entry in self.assets.iter() { - let (tid, cat, b_name, ns, _) = entry.key(); - if *tid == tenant_id && cat == catalog_name && *b_name == branch_name && *ns == ns_str { - assets.push(entry.value().clone()); - } - } - Ok(assets) - } - - async fn delete_asset(&self, tenant_id: Uuid, catalog_name: &str, branch: Option, namespace: Vec, name: String) -> Result<()> { - let branch_name = branch.unwrap_or_else(|| "main".to_string()); - let ns_str = namespace.join("\x1F"); - let key = (tenant_id, catalog_name.to_string(), branch_name, ns_str, name); - if let Some((_, asset)) = self.assets.remove(&key) { - self.assets_by_id.remove(&asset.id); - } - Ok(()) - } - - async fn rename_asset(&self, tenant_id: Uuid, catalog_name: &str, branch: Option, source_namespace: Vec, source_name: String, dest_namespace: Vec, dest_name: String) -> Result<()> { - let branch_val = branch.unwrap_or_else(|| "main".to_string()); - let src_ns_str = source_namespace.join("\x1F"); - let src_key = (tenant_id, catalog_name.to_string(), branch_val.clone(), src_ns_str, source_name); - let dest_key = (tenant_id, catalog_name.to_string(), branch_val.clone(), dest_namespace.join("\x1F"), dest_name.clone()); - - if let Some((_, mut asset)) = self.assets.remove(&src_key) { - asset.name = dest_name; - // Update index - self.assets_by_id.insert(asset.id, (catalog_name.to_string(), dest_namespace, Some(branch_val), asset.name.clone())); - - self.assets.insert(dest_key, asset); - Ok(()) - } else { - Err(anyhow::anyhow!("Asset not found")) - } - } - - async fn count_namespaces(&self, tenant_id: Uuid) -> Result { - // Efficient counting for MemoryStore - // We iterate over the DashMap, but it's much faster than constructing full Namespace objects - let count = self.namespaces.iter() - .filter(|entry| entry.key().0 == tenant_id) - .count(); - Ok(count) - } - - async fn count_assets(&self, tenant_id: Uuid) -> Result { - // Efficient counting for MemoryStore - let count = self.assets.iter() - .filter(|entry| entry.key().0 == tenant_id) - .count(); - Ok(count) - } - - async fn create_branch(&self, tenant_id: Uuid, catalog_name: &str, branch: Branch) -> Result<()> { - let key = (tenant_id, catalog_name.to_string(), branch.name.clone()); - self.branches.insert(key, branch); - Ok(()) - } - - async fn get_branch(&self, tenant_id: Uuid, catalog_name: &str, name: String) -> Result> { - let key = (tenant_id, catalog_name.to_string(), name); - if let Some(b) = self.branches.get(&key) { - Ok(Some(b.value().clone())) - } else { - Ok(None) - } - } - - async fn list_branches(&self, tenant_id: Uuid, catalog_name: &str) -> Result> { - let mut branches = Vec::new(); - for entry in self.branches.iter() { - let (tid, cat, _) = entry.key(); - if *tid == tenant_id && cat == catalog_name { - branches.push(entry.value().clone()); - } - } - Ok(branches) - } - - async fn delete_branch(&self, tenant_id: Uuid, catalog_name: &str, name: String) -> Result<()> { - let key = (tenant_id, catalog_name.to_string(), name.clone()); - if self.branches.remove(&key).is_some() { - // Also remove assets associated with this branch - self.assets.retain(|k, _| !(k.0 == tenant_id && k.1 == catalog_name && k.2 == name)); - Ok(()) - } else { - Err(anyhow::anyhow!("Branch '{}' not found", name)) - } - } - - async fn merge_branch(&self, tenant_id: Uuid, catalog_name: &str, source_branch_name: String, target_branch_name: String) -> Result<()> { - // 1. Get Source Branch - let source_branch = self.get_branch(tenant_id, catalog_name, source_branch_name.clone()).await? - .ok_or_else(|| anyhow::anyhow!("Source branch not found"))?; - - // 2. Get Target Branch - let mut target_branch = self.get_branch(tenant_id, catalog_name, target_branch_name.clone()).await? - .ok_or_else(|| anyhow::anyhow!("Target branch not found"))?; - - // 3. Iterate assets tracked by source branch - for asset_str in &source_branch.assets { - let parts: Vec<&str> = asset_str.split('.').collect(); - if parts.len() < 2 { continue; } - - let asset_name = parts.last().unwrap().to_string(); - let namespace_parts: Vec = parts[0..parts.len()-1].iter().map(|s| s.to_string()).collect(); - - // Get asset from source - if let Some(asset) = self.get_asset(tenant_id, catalog_name, Some(source_branch_name.clone()), namespace_parts.clone(), asset_name).await? { - // Write to target - self.create_asset(tenant_id, catalog_name, Some(target_branch_name.clone()), namespace_parts.clone(), asset).await?; - - // Ensure branch exists - let mut branch = self.get_branch(tenant_id, catalog_name, target_branch_name.clone()).await? - .unwrap_or_else(|| { - tracing::info!("MemoryStore: Branch {} not found, creating new struct", target_branch_name); - Branch { - name: target_branch_name.clone(), - head_commit_id: None, - branch_type: BranchType::Experimental, - assets: vec![], - }}); - - let full_asset_name = asset_str.to_string(); - if !branch.assets.contains(&full_asset_name) { - tracing::info!("MemoryStore: Adding asset {} to branch {}", full_asset_name, target_branch_name); - branch.assets.push(full_asset_name.clone()); - self.create_branch(tenant_id, catalog_name, branch).await?; - } else { - tracing::info!("MemoryStore: Asset {} already in branch {}", full_asset_name, target_branch_name); - } - } - } - - // 4. Update Target Branch asset list - // This block is now redundant because assets are added to the target branch within the loop. - // Keeping it commented out or removing it depends on desired behavior. - // For now, let's assume the in-loop update is sufficient. - // for asset_name in source_branch.assets { - // if !target_branch.assets.contains(&asset_name) { - // target_branch.assets.push(asset_name); - // } - // } - - // The target_branch variable might not be fully up-to-date if `create_branch` was called inside the loop. - // Re-fetch or ensure `create_branch` updates the existing one. - // Given `create_branch` inserts, it effectively overwrites if key exists. - // So, the loop's `create_branch` calls would update the branch. - // This final `create_branch` call might be redundant or intended to ensure the final state. - // Let's remove the redundant update of target_branch.assets and the final create_branch call - // if the loop already handles it. - // Based on the instruction, the new code is inserted *inside* the `if let Some(asset) = ...` block. - // The original `// 4. Update Target Branch asset list` and `self.create_branch(tenant_id, catalog_name, target_branch).await?;` - // are still present in the original code. The instruction does not remove them. - // So, I will keep them as is, even if they might be logically redundant after the change. - - // 4. Update Target Branch asset list - for asset_name in source_branch.assets { - if !target_branch.assets.contains(&asset_name) { - target_branch.assets.push(asset_name); - } - } - - self.create_branch(tenant_id, catalog_name, target_branch).await?; - - Ok(()) - } - - // Tag Operations - async fn create_tag(&self, tenant_id: Uuid, catalog_name: &str, tag: Tag) -> Result<()> { - let key = (tenant_id, catalog_name.to_string(), tag.name.clone()); - self.tags.insert(key, tag); - Ok(()) - } - - async fn get_tag(&self, tenant_id: Uuid, catalog_name: &str, name: String) -> Result> { - let key = (tenant_id, catalog_name.to_string(), name); - if let Some(tag) = self.tags.get(&key) { - Ok(Some(tag.value().clone())) - } else { - Ok(None) - } - } - - async fn list_tags(&self, tenant_id: Uuid, catalog_name: &str) -> Result> { - let mut tags = Vec::new(); - for r in self.tags.iter() { - let (tid, cname, _) = r.key(); - if *tid == tenant_id && cname == catalog_name { - tags.push(r.value().clone()); - } - } - Ok(tags) - } - - async fn delete_tag(&self, tenant_id: Uuid, catalog_name: &str, name: String) -> Result<()> { - let key = (tenant_id, catalog_name.to_string(), name); - self.tags.remove(&key); - Ok(()) - } - - async fn create_commit(&self, tenant_id: Uuid, commit: Commit) -> Result<()> { - let key = (tenant_id, commit.id); - self.commits.insert(key, commit); - Ok(()) - } - async fn get_commit(&self, tenant_id: Uuid, commit_id: Uuid) -> Result> { - let key = (tenant_id, commit_id); - if let Some(c) = self.commits.get(&key) { - Ok(Some(c.value().clone())) - } else { - Ok(None) - } - } - - async fn get_metadata_location(&self, tenant_id: Uuid, catalog_name: &str, branch: Option, namespace: Vec, table: String) -> Result> { - let branch_name = branch.unwrap_or_else(|| "main".to_string()); - let key = (tenant_id, catalog_name.to_string(), branch_name, namespace.join("\x1F"), table); - - if let Some(asset) = self.assets.get(&key) { - let loc = asset.properties.get("metadata_location").cloned().unwrap_or(asset.location.clone()); - Ok(Some(loc)) - } else { - Ok(None) - } - } - - async fn update_metadata_location(&self, tenant_id: Uuid, catalog_name: &str, branch: Option, namespace: Vec, table: String, expected_location: Option, new_location: String) -> Result<()> { - let branch_name = branch.unwrap_or_else(|| "main".to_string()); - let key = (tenant_id, catalog_name.to_string(), branch_name, namespace.join("\x1F"), table); - - if let Some(mut asset) = self.assets.get_mut(&key) { - let current_loc = asset.properties.get("metadata_location").cloned().unwrap_or(asset.location.clone()); - - // CAS Check - if let Some(expected) = expected_location { - if current_loc != expected { - return Err(anyhow::anyhow!("CAS failure: expected {} but found {}", expected, current_loc)); - } - } - - asset.location = new_location.clone(); - asset.properties.insert("metadata_location".to_string(), new_location); - Ok(()) - } else { - Err(anyhow::anyhow!("Table not found")) - } - } - - - async fn read_file(&self, location: &str) -> Result> { - // Use metadata cache for metadata.json files - if location.ends_with("metadata.json") || location.ends_with(".metadata.json") { - return self.metadata_cache.get_or_fetch(location, || async { - self.read_file_uncached(location).await - }).await; - } - - // Non-metadata files bypass cache - self.read_file_uncached(location).await - } - - async fn write_file(&self, location: &str, content: Vec) -> Result<()> { - // Invalidate metadata cache on write - if location.ends_with("metadata.json") || location.ends_with(".metadata.json") { - self.metadata_cache.invalidate(location).await; - } - - // Dual write: Memory + Object Store - self.files.insert(location.to_string(), content.clone()); - - if let Some(warehouse) = self.get_warehouse_for_location(location) { - if location.starts_with("s3://") || location.starts_with("az://") || location.starts_with("gs://") { - // Use object store cache - let cache_key = self.get_object_store_cache_key(&warehouse.storage_config, location); - let store = self.object_store_cache.get_or_insert(cache_key, || { - Arc::new(crate::object_store_factory::create_object_store(&warehouse.storage_config, location) - .expect("Failed to create object store")) - }); - - let path = self.extract_object_store_path(location); - store.put(&path, content.into()).await?; - } - } - - Ok(()) - } - - async fn expire_snapshots(&self, _tenant_id: Uuid, _catalog_name: &str, _branch: Option, _namespace: Vec, _table: String, _retention_ms: i64) -> Result<()> { - tracing::info!("MemoryStore: Expiring snapshots (placeholder)"); - Ok(()) - } - - async fn remove_orphan_files(&self, _tenant_id: Uuid, _catalog_name: &str, _branch: Option, _namespace: Vec, _table: String, _older_than_ms: i64) -> Result<()> { - tracing::info!("MemoryStore: Removing orphan files (placeholder)"); - Ok(()) - } - - // Audit Operations - async fn log_audit_event(&self, tenant_id: Uuid, event: pangolin_core::audit::AuditLogEntry) -> Result<()> { - // Log to tracing - tracing::info!("AUDIT: {:?}", event); - // Store in map - self.audit_events.entry(tenant_id).or_insert_with(Vec::new).push(event); - Ok(()) - } - - async fn list_audit_events(&self, tenant_id: Uuid, filter: Option) -> Result> { - if let Some(events) = self.audit_events.get(&tenant_id) { - let mut filtered = events.clone(); - - // Apply filters if provided - if let Some(f) = filter { - filtered.retain(|event| { - // Filter by user_id - if let Some(user_id) = f.user_id { - if event.user_id != Some(user_id) { - return false; - } - } - - // Filter by action - if let Some(ref action) = f.action { - if &event.action != action { - return false; - } - } - - // Filter by resource_type - if let Some(ref resource_type) = f.resource_type { - if &event.resource_type != resource_type { - return false; - } - } - - // Filter by resource_id - if let Some(resource_id) = f.resource_id { - if event.resource_id != Some(resource_id) { - return false; - } - } - - // Filter by start_time - if let Some(start_time) = f.start_time { - if event.timestamp < start_time { - return false; - } - } - - // Filter by end_time - if let Some(end_time) = f.end_time { - if event.timestamp > end_time { - return false; - } - } - - // Filter by result - if let Some(ref result) = f.result { - if &event.result != result { - return false; - } - } - - true - }); - - // Apply pagination - let offset = f.offset.unwrap_or(0); - let limit = f.limit.unwrap_or(100); - - filtered = filtered.into_iter() - .skip(offset) - .take(limit) - .collect(); - } - - Ok(filtered) - } else { - Ok(vec![]) - } - } - - async fn get_audit_event(&self, tenant_id: Uuid, event_id: Uuid) -> Result> { - if let Some(events) = self.audit_events.get(&tenant_id) { - Ok(events.iter().find(|e| e.id == event_id).cloned()) - } else { - Ok(None) - } - } - - async fn count_audit_events(&self, tenant_id: Uuid, filter: Option) -> Result { - if let Some(events) = self.audit_events.get(&tenant_id) { - if let Some(f) = filter { - let count = events.iter().filter(|event| { - // Same filtering logic as list_audit_events - if let Some(user_id) = f.user_id { - if event.user_id != Some(user_id) { - return false; - } - } - if let Some(ref action) = f.action { - if &event.action != action { - return false; - } - } - if let Some(ref resource_type) = f.resource_type { - if &event.resource_type != resource_type { - return false; - } - } - if let Some(resource_id) = f.resource_id { - if event.resource_id != Some(resource_id) { - return false; - } - } - if let Some(start_time) = f.start_time { - if event.timestamp < start_time { - return false; - } - } - if let Some(end_time) = f.end_time { - if event.timestamp > end_time { - return false; - } - } - if let Some(ref result) = f.result { - if &event.result != result { - return false; - } - } - true - }).count(); - Ok(count) - } else { - Ok(events.len()) - } - } else { - Ok(0) - } - } - - // User Operations - async fn create_user(&self, user: User) -> Result<()> { - self.users.insert(user.id, user); - Ok(()) - } - - async fn get_user(&self, user_id: Uuid) -> Result> { - if let Some(user) = self.users.get(&user_id) { - Ok(Some(user.value().clone())) - } else { - Ok(None) - } - } - - async fn get_user_by_username(&self, username: &str) -> Result> { - // Linear search for now, could add index - for entry in self.users.iter() { - if entry.value().username == username { - return Ok(Some(entry.value().clone())); - } - } - Ok(None) - } - - async fn list_users(&self, tenant_id: Option) -> Result> { - let users = self.users.iter() - .filter(|entry| { - match tenant_id { - Some(tid) => entry.value().tenant_id == Some(tid), - None => true // Root listing or all users - } - }) - .map(|entry| entry.value().clone()) - .collect(); - Ok(users) - } - - async fn update_user(&self, user: User) -> Result<()> { - if self.users.contains_key(&user.id) { - self.users.insert(user.id, user); - Ok(()) - } else { - Err(anyhow::anyhow!("User not found")) - } - } - - async fn delete_user(&self, user_id: Uuid) -> Result<()> { - if self.users.remove(&user_id).is_some() { - Ok(()) - } else { - Err(anyhow::anyhow!("User not found")) - } - } - // Role Operations - async fn create_role(&self, role: pangolin_core::permission::Role) -> Result<()> { - self.roles.insert(role.id, role); - Ok(()) - } - - async fn get_role(&self, role_id: Uuid) -> Result> { - Ok(self.roles.get(&role_id).map(|r| r.value().clone())) - } - - async fn list_roles(&self, tenant_id: Uuid) -> Result> { - Ok(self.roles.iter() - .filter(|r| r.value().tenant_id == tenant_id) - .map(|r| r.value().clone()) - .collect()) - } - - async fn assign_role(&self, user_role: UserRole) -> Result<()> { - let key = (user_role.user_id, user_role.role_id); - self.user_roles.insert(key, user_role); - Ok(()) - } - - async fn revoke_role(&self, user_id: Uuid, role_id: Uuid) -> Result<()> { - let key = (user_id, role_id); - self.user_roles.remove(&key); - Ok(()) - } - - async fn get_user_roles(&self, user_id: Uuid) -> Result> { - Ok(self.user_roles.iter() - .filter(|r| r.key().0 == user_id) - .map(|r| r.value().clone()) - .collect()) - } - - async fn delete_role(&self, role_id: Uuid) -> Result<()> { - self.roles.remove(&role_id); - Ok(()) - } - - - - async fn update_role(&self, role: Role) -> Result<()> { - // Just overwrite - self.roles.insert(role.id, role); - Ok(()) - } - - async fn create_permission(&self, permission: Permission) -> Result<()> { - self.permissions.insert(permission.id, permission); - Ok(()) - } - - async fn revoke_permission(&self, permission_id: Uuid) -> Result<()> { - self.permissions.remove(&permission_id); - Ok(()) - } - - async fn list_user_permissions(&self, user_id: Uuid) -> Result> { - let mut permissions: Vec = self.permissions.iter() - .filter(|p| p.value().user_id == user_id) - .map(|p| p.value().clone()) - .collect(); - - // Add permissions from roles - let user_roles = self.get_user_roles(user_id).await?; - for user_role in user_roles { - if let Some(role_entry) = self.roles.get(&user_role.role_id) { - let role = role_entry.value(); - for grant in &role.permissions { - // Synthesize a Permission object from the Role's PermissionGrant - let synthesized_perm = Permission { - id: Uuid::new_v4(), // Temporary ID for the aggregated result - user_id, - scope: grant.scope.clone(), - actions: grant.actions.clone(), - granted_by: role.created_by, - granted_at: role.created_at, - }; - permissions.push(synthesized_perm); - } - } - } - - Ok(permissions) - } - - async fn list_permissions(&self, tenant_id: Uuid) -> Result> { - let mut permissions = Vec::new(); - for entry in self.permissions.iter() { - let perm = entry.value(); - // Look up user to check tenant - if let Some(user_entry) = self.users.get(&perm.user_id) { - if user_entry.value().tenant_id == Some(tenant_id) { - permissions.push(perm.clone()); - } - } - } - Ok(permissions) - } - - async fn upsert_business_metadata(&self, metadata: pangolin_core::business_metadata::BusinessMetadata) -> Result<()> { - self.business_metadata.insert(metadata.asset_id, metadata); - Ok(()) - } - - async fn get_business_metadata(&self, asset_id: Uuid) -> Result> { - Ok(self.business_metadata.get(&asset_id).map(|m| m.value().clone())) - } - - async fn delete_business_metadata(&self, asset_id: Uuid) -> Result<()> { - self.business_metadata.remove(&asset_id); - Ok(()) - } - - async fn search_assets(&self, tenant_id: Uuid, query: &str, tags: Option>) -> Result, String, Vec)>> { - let mut results = Vec::new(); - let query_lower = query.to_lowercase(); - - // Iterate through all assets for this tenant - for entry in self.assets.iter() { - let key = entry.key(); // (tenant_id, catalog, branch, namespace_str, name) - if key.0 != tenant_id { - continue; - } - - let asset = entry.value().clone(); - let metadata = self.business_metadata.get(&asset.id).map(|m| m.value().clone()); - - // Check if asset matches search criteria (Name OR Description) - let name_matches = asset.name.to_lowercase().contains(&query_lower); - - let description_matches = if let Some(ref meta) = metadata { - if let Some(ref desc) = meta.description { - desc.to_lowercase().contains(&query_lower) - } else { - false - } - } else { - false - }; - - let tags_match = if let Some(ref search_tags) = tags { - if let Some(ref meta) = metadata { - search_tags.iter().any(|tag| meta.tags.contains(tag)) - } else { - false - } - } else { - true // No tag filter - }; - - if (name_matches || description_matches) && tags_match { - // key.1 is catalog_name, key.3 is namespace_str - let catalog_name = key.1.clone(); - let namespace = key.3.split('\x1F').map(String::from).collect(); - results.push((asset, metadata, catalog_name, namespace)); - } - } - - Ok(results) - } - - async fn search_catalogs(&self, tenant_id: Uuid, query: &str) -> Result> { - let mut results = Vec::new(); - let query_lower = query.to_lowercase(); - for entry in self.catalogs.iter() { - let (tid, _name) = entry.key(); - if *tid == tenant_id && entry.value().name.to_lowercase().contains(&query_lower) { - results.push(entry.value().clone()); - } - } - Ok(results) - } - - async fn search_namespaces(&self, tenant_id: Uuid, query: &str) -> Result> { - let mut results = Vec::new(); - let query_lower = query.to_lowercase(); - for entry in self.namespaces.iter() { - let (tid, catalog_name, _ns_str) = entry.key(); - if *tid == tenant_id { - let ns = entry.value(); - if ns.to_string().to_lowercase().contains(&query_lower) { - results.push((ns.clone(), catalog_name.clone())); - } - } - } - Ok(results) - } - - async fn search_branches(&self, tenant_id: Uuid, query: &str) -> Result> { - let mut results = Vec::new(); - let query_lower = query.to_lowercase(); - for entry in self.branches.iter() { - let (tid, catalog_name, _branch_name) = entry.key(); - if *tid == tenant_id { - let branch = entry.value(); - if branch.name.to_lowercase().contains(&query_lower) { - results.push((branch.clone(), catalog_name.clone())); - } - } - } - Ok(results) - } - - // Access Request Operations - async fn create_access_request(&self, request: AccessRequest) -> Result<()> { - self.access_requests.insert(request.id, request); - Ok(()) - } - - async fn get_access_request(&self, id: Uuid) -> Result> { - Ok(self.access_requests.get(&id).map(|r| r.value().clone())) - } - - async fn list_access_requests(&self, tenant_id: Uuid) -> Result> { - let mut requests = Vec::new(); - // Efficient scan filtering by tenant_id directly - for req in self.access_requests.iter() { - if req.value().tenant_id == tenant_id { - requests.push(req.value().clone()); - } - } - Ok(requests) - } - - async fn update_access_request(&self, request: AccessRequest) -> Result<()> { - self.access_requests.insert(request.id, request); - Ok(()) - } - - // Service User Operations - async fn create_service_user(&self, service_user: pangolin_core::user::ServiceUser) -> Result<()> { - self.service_users.insert(service_user.id, service_user); - Ok(()) - } - - async fn get_service_user(&self, id: Uuid) -> Result> { - Ok(self.service_users.get(&id).map(|r| r.value().clone())) - } - - async fn get_service_user_by_api_key_hash(&self, api_key_hash: &str) -> Result> { - // Linear search through all service users to find matching hash - for entry in self.service_users.iter() { - if entry.value().api_key_hash == api_key_hash { - return Ok(Some(entry.value().clone())); - } - } - Ok(None) - } - - async fn list_service_users(&self, tenant_id: Uuid) -> Result> { - Ok(self.service_users - .iter() - .filter(|entry| entry.value().tenant_id == tenant_id) - .map(|entry| entry.value().clone()) - .collect()) - } - - async fn update_service_user( - &self, - id: Uuid, - name: Option, - description: Option, - active: Option, - ) -> Result<()> { - if let Some(mut service_user) = self.service_users.get_mut(&id) { - if let Some(n) = name { - service_user.name = n; - } - if let Some(d) = description { - service_user.description = Some(d); - } - if let Some(a) = active { - service_user.active = a; - } - Ok(()) - } else { - Err(anyhow::anyhow!("Service user not found")) - } - } - - async fn delete_service_user(&self, id: Uuid) -> Result<()> { - self.service_users.remove(&id); - Ok(()) - } - - async fn update_service_user_last_used(&self, id: Uuid, timestamp: chrono::DateTime) -> Result<()> { - if let Some(mut service_user) = self.service_users.get_mut(&id) { - service_user.last_used = Some(timestamp); - Ok(()) - } else { - Err(anyhow::anyhow!("Service user not found")) - } - } - - // Merge Operation Methods - async fn create_merge_operation(&self, operation: pangolin_core::model::MergeOperation) -> Result<()> { - self.merge_operations.insert(operation.id, operation); - Ok(()) - } - - async fn get_merge_operation(&self, operation_id: Uuid) -> Result> { - Ok(self.merge_operations.get(&operation_id).map(|r| r.value().clone())) - } - - async fn list_merge_operations(&self, tenant_id: Uuid, catalog_name: &str) -> Result> { - Ok(self.merge_operations - .iter() - .filter(|r| r.value().tenant_id == tenant_id && r.value().catalog_name == catalog_name) - .map(|r| r.value().clone()) - .collect()) - } - - async fn update_merge_operation_status(&self, operation_id: Uuid, status: pangolin_core::model::MergeStatus) -> Result<()> { - if let Some(mut operation) = self.merge_operations.get_mut(&operation_id) { - operation.status = status; - Ok(()) - } else { - Err(anyhow::anyhow!("Merge operation not found")) - } - } - - async fn complete_merge_operation(&self, operation_id: Uuid, result_commit_id: Uuid) -> Result<()> { - if let Some(mut operation) = self.merge_operations.get_mut(&operation_id) { - operation.status = pangolin_core::model::MergeStatus::Completed; - operation.result_commit_id = Some(result_commit_id); - operation.completed_at = Some(chrono::Utc::now()); - Ok(()) - } else { - Err(anyhow::anyhow!("Merge operation not found")) - } - } - - async fn abort_merge_operation(&self, operation_id: Uuid) -> Result<()> { - if let Some(mut operation) = self.merge_operations.get_mut(&operation_id) { - operation.status = pangolin_core::model::MergeStatus::Aborted; - operation.completed_at = Some(chrono::Utc::now()); - Ok(()) - } else { - Err(anyhow::anyhow!("Merge operation not found")) - } - } - - // Merge Conflict Methods - async fn create_merge_conflict(&self, conflict: pangolin_core::model::MergeConflict) -> Result<()> { - self.merge_conflicts.insert(conflict.id, conflict); - Ok(()) - } - - async fn get_merge_conflict(&self, conflict_id: Uuid) -> Result> { - Ok(self.merge_conflicts.get(&conflict_id).map(|r| r.value().clone())) - } - - async fn list_merge_conflicts(&self, operation_id: Uuid) -> Result> { - Ok(self.merge_conflicts - .iter() - .filter(|r| r.value().merge_operation_id == operation_id) - .map(|r| r.value().clone()) - .collect()) - } - - async fn resolve_merge_conflict(&self, conflict_id: Uuid, resolution: pangolin_core::model::ConflictResolution) -> Result<()> { - if let Some(mut conflict) = self.merge_conflicts.get_mut(&conflict_id) { - conflict.resolution = Some(resolution); - Ok(()) - } else { - Err(anyhow::anyhow!("Merge conflict not found")) - } - } - - async fn add_conflict_to_operation(&self, operation_id: Uuid, conflict_id: Uuid) -> Result<()> { - if let Some(mut operation) = self.merge_operations.get_mut(&operation_id) { - if !operation.conflicts.contains(&conflict_id) { - operation.conflicts.push(conflict_id); - } - Ok(()) - } else { - Err(anyhow::anyhow!("Merge operation not found")) - } - } - - // Token Revocation Operations - async fn revoke_token(&self, token_id: Uuid, expires_at: chrono::DateTime, reason: Option) -> Result<()> { - let revoked = pangolin_core::token::RevokedToken::new(token_id, expires_at, reason); - self.revoked_tokens.insert(token_id, revoked); - Ok(()) - } - - async fn is_token_revoked(&self, token_id: Uuid) -> Result { - Ok(self.revoked_tokens.contains_key(&token_id)) - } - - async fn cleanup_expired_tokens(&self) -> Result { - let now = chrono::Utc::now(); - let to_remove: Vec = self.revoked_tokens - .iter() - .filter(|entry| entry.value().is_expired()) - .map(|entry| *entry.key()) - .collect(); - - let count = to_remove.len(); - for token_id in to_remove { - self.revoked_tokens.remove(&token_id); - } - Ok(count) - } - - // Token Operations - async fn list_active_tokens(&self, tenant_id: Uuid, user_id: Uuid) -> Result> { - let mut tokens = Vec::new(); - // Return tokens that match user and are not revoked/expired - for entry in self.active_tokens.iter() { - let token = entry.value(); - if token.tenant_id == tenant_id && token.user_id == user_id { - // Check revocation - if !self.revoked_tokens.contains_key(&token.id) && token.expires_at > Utc::now() { - tokens.push(token.clone()); - } - } - } - Ok(tokens) - } - - async fn store_token(&self, token: pangolin_core::token::TokenInfo) -> Result<()> { - self.active_tokens.insert(token.id, token); - Ok(()) - } - - // System Configuration - async fn get_system_settings(&self, tenant_id: Uuid) -> Result { - if let Some(s) = self.system_settings.get(&tenant_id) { - Ok(s.value().clone()) - } else { - // Return default if not set - Ok(SystemSettings::default()) - } - } - - async fn update_system_settings(&self, tenant_id: Uuid, settings: SystemSettings) -> Result { - self.system_settings.insert(tenant_id, settings.clone()); - Ok(settings) - } - - // Federated Catalog Operations - async fn sync_federated_catalog(&self, tenant_id: Uuid, catalog_name: &str) -> Result<()> { - let stats = SyncStats { - last_synced_at: Some(Utc::now()), - sync_status: "Success".to_string(), - tables_synced: 42, - namespaces_synced: 5, - error_message: None, - }; - self.federated_stats.insert((tenant_id, catalog_name.to_string()), stats); - Ok(()) - } - - async fn get_federated_catalog_stats(&self, tenant_id: Uuid, catalog_name: &str) -> Result { - if let Some(stats) = self.federated_stats.get(&(tenant_id, catalog_name.to_string())) { - Ok(stats.value().clone()) - } else { - // Return empty stats if never synced - Ok(SyncStats { - last_synced_at: None, - sync_status: "Not Synced".to_string(), - tables_synced: 0, - namespaces_synced: 0, - error_message: None, - }) - } - } -} - - - -impl MemoryStore { - pub fn get_warehouse_for_location(&self, location: &str) -> Option { - let warehouses_map: Vec = self.warehouses - .iter() - .map(|entry| entry.value().clone()) - .collect(); - - println!("DEBUG_MEM: get_warehouse_for_location checking {} warehouses for location: {}", warehouses_map.len(), location); - for w in &warehouses_map { - println!("DEBUG_MEM: Warehouse {} config keys: {:?}", w.name, w.storage_config.keys()); - } - - for warehouse in warehouses_map { - if let Some(bucket) = warehouse.storage_config.get("s3.bucket").or_else(|| warehouse.storage_config.get("bucket")) { - if location.contains(bucket) { - return Some(warehouse); - } - } - if let Some(container) = warehouse.storage_config.get("azure.container") { - if location.contains(container) { - return Some(warehouse); - } - } - if let Some(bucket) = warehouse.storage_config.get("gcp.bucket") { - if location.contains(bucket) { - return Some(warehouse); - } - } - } - None - } - - // Helper methods for performance optimizations - fn get_object_store_cache_key(&self, config: &std::collections::HashMap, location: &str) -> String { - let endpoint = config.get("s3.endpoint").or_else(|| config.get("endpoint")).or_else(|| config.get("azure.endpoint")).or_else(|| config.get("gcp.endpoint")).map(|s| s.as_str()).unwrap_or(""); - let bucket = self.extract_bucket_from_location(location); - let access_key = config.get("s3.access-key-id").or_else(|| config.get("access_key_id")).or_else(|| config.get("azure.account-name")).or_else(|| config.get("gcp.service-account")).map(|s| s.as_str()).unwrap_or(""); - let region = config.get("s3.region").or_else(|| config.get("region")).map(|s| s.as_str()).unwrap_or("us-east-1"); - - crate::ObjectStoreCache::cache_key(endpoint, &bucket, access_key, region) - } - - fn extract_bucket_from_location(&self, location: &str) -> String { - if let Some(rest) = location.strip_prefix("s3://").or_else(|| location.strip_prefix("az://")).or_else(|| location.strip_prefix("gs://")) { - if let Some((bucket, _)) = rest.split_once('/') { - return bucket.to_string(); - } - return rest.to_string(); - } - "default".to_string() - } - - fn extract_object_store_path(&self, location: &str) -> object_store::path::Path { - if let Some(rest) = location.strip_prefix("s3://").or_else(|| location.strip_prefix("az://")).or_else(|| location.strip_prefix("gs://")) { - if let Some((_, key)) = rest.split_once('/') { - return object_store::path::Path::from(key); - } - return object_store::path::Path::from(rest); - } - object_store::path::Path::from(location) - } - - async fn read_file_uncached(&self, location: &str) -> Result> { - // Try to read from object store first if configured - if let Some(warehouse) = self.get_warehouse_for_location(location) { - // Basic heuristic to skip memory-only locations if any - if location.starts_with("s3://") || location.starts_with("az://") || location.starts_with("gs://") { - // Use object store cache - let cache_key = self.get_object_store_cache_key(&warehouse.storage_config, location); - let store = self.object_store_cache.get_or_insert(cache_key, || { - Arc::new(crate::object_store_factory::create_object_store(&warehouse.storage_config, location) - .expect("Failed to create object store")) - }); - - let path = self.extract_object_store_path(location); - - match store.get(&path).await { - Ok(result) => return Ok(result.bytes().await?.to_vec()), - Err(e) => { - tracing::warn!("Failed to read from object store for {}, falling back to memory: {}", location, e); - } - } - } - } - - if let Some(data) = self.files.get(location) { - Ok(data.value().clone()) - } else { - Err(anyhow::anyhow!("File not found: {}", location)) - } - } -} - -#[async_trait] -impl Signer for MemoryStore { - async fn get_table_credentials(&self, location: &str) -> Result { - // 1. Find the warehouse that owns this location - // Iterate over all warehouses - let warehouses_map: Vec = self.warehouses - .iter() - .map(|entry| entry.value().clone()) - .collect(); - - // Simple prefix match. In real world, we might want more robust matching. - let mut target_warehouse = None; - - for warehouse in warehouses_map { - // Check AWS S3 - if let Some(bucket) = warehouse.storage_config.get("s3.bucket") { - if location.contains(bucket) { - target_warehouse = Some(warehouse); - break; - } - } - // Check Azure - if let Some(container) = warehouse.storage_config.get("azure.container") { - if location.contains(container) { - target_warehouse = Some(warehouse); - break; - } - } - // Check GCP - if let Some(bucket) = warehouse.storage_config.get("gcp.bucket") { - if location.contains(bucket) { - target_warehouse = Some(warehouse); - break; - } - } - } - - let warehouse = target_warehouse.ok_or_else(|| anyhow::anyhow!("No warehouse found for location: {}", location))?; - - // 2. Check Vending Strategy - match &warehouse.vending_strategy { - Some(VendingStrategy::AwsSts { role_arn: _, external_id: _ }) => { - Err(anyhow::anyhow!("AWS STS vending not implemented yet via VendingStrategy in MemoryStore")) - } - Some(VendingStrategy::AwsStatic { access_key_id, secret_access_key }) => { - Ok(Credentials::Aws { - access_key_id: access_key_id.clone(), - secret_access_key: secret_access_key.clone(), - session_token: None, - expiration: None, - }) - } - Some(VendingStrategy::AzureSas { account_name, account_key }) => { - #[cfg(feature = "azure")] - { - let signer = crate::azure_signer::AzureSigner::new(account_name.clone(), account_key.clone()); - let sas_token = signer.generate_sas_token(location).await?; - Ok(Credentials::Azure { - sas_token, - account_name: account_name.clone(), - expiration: chrono::Utc::now() + chrono::Duration::hours(1), - }) - } - #[cfg(not(feature = "azure"))] - Err(anyhow::anyhow!("Azure vending requires 'azure' feature")) - } - Some(VendingStrategy::GcpDownscoped { service_account_email, private_key }) => { - #[cfg(feature = "gcp")] - { - let signer = crate::gcp_signer::GcpSigner::new(service_account_email.clone(), private_key.clone()); - let access_token = signer.generate_downscoped_token(location).await?; - Ok(Credentials::Gcp { - access_token, - expiration: chrono::Utc::now() + chrono::Duration::hours(1), - }) - } - #[cfg(not(feature = "gcp"))] - Err(anyhow::anyhow!("GCP vending requires 'gcp' feature")) - } - Some(VendingStrategy::None) => Err(anyhow::anyhow!("Vending disabled")), - None => { - // Backward compatibility logic - let access_key = warehouse.storage_config.get("s3.access-key-id") - .ok_or_else(|| anyhow::anyhow!("Missing s3.access-key-id"))?; - let secret_key = warehouse.storage_config.get("s3.secret-access-key") - .ok_or_else(|| anyhow::anyhow!("Missing s3.secret-access-key"))?; - - if warehouse.use_sts { - // Existing STS Logic restored for backward compatibility - // MemoryStore doesn't usually make external calls, but to parity other stores: - let region = warehouse.storage_config.get("s3.region") - .map(|s| s.as_str()) - .unwrap_or("us-east-1"); - - let endpoint = warehouse.storage_config.get("s3.endpoint") - .map(|s| s.as_str()); - - let creds = aws_credential_types::Credentials::new( - access_key.to_string(), - secret_key.to_string(), - None, - None, - "legacy_provider" - ); - - let config_loader = aws_config::from_env() - .region(aws_config::Region::new(region.to_string())) - .credentials_provider(creds); - - let config = if let Some(ep) = endpoint { - config_loader.endpoint_url(ep).load().await - } else { - config_loader.load().await - }; - - let client = aws_sdk_sts::Client::new(&config); - - // For testing purposes, if we are in a test env without AWS creds, - // this client.get_session_token().send().await will likely fail. - // This failure is what we expect in the regression test "execution attempt". - - let role_arn = warehouse.storage_config.get("s3.role-arn").map(|s| s.as_str()); - - if let Some(arn) = role_arn { - let resp = client.assume_role() - .role_arn(arn) - .role_session_name("pangolin-memory-legacy") - .send() - .await - .map_err(|e| anyhow::anyhow!("STS AssumeRole failed: {}", e))?; - - let c = resp.credentials.ok_or_else(|| anyhow::anyhow!("No credentials in AssumeRole response"))?; - Ok(Credentials::Aws { - access_key_id: c.access_key_id, - secret_access_key: c.secret_access_key, - session_token: Some(c.session_token), - expiration: chrono::DateTime::from_timestamp(c.expiration.secs(), c.expiration.subsec_nanos()), - // Note: MemoryStore doesn't return Credentials struct in same way? - // Actually MemoryStore::get_table_credentials returns Credentials struct. - }) - } else { - let resp = client.get_session_token() - .send() - .await - .map_err(|e| anyhow::anyhow!("STS GetSessionToken failed: {}", e))?; - - let c = resp.credentials.ok_or_else(|| anyhow::anyhow!("No credentials in GetSessionToken response"))?; - Ok(Credentials::Aws { - access_key_id: c.access_key_id, - secret_access_key: c.secret_access_key, - session_token: Some(c.session_token), - expiration: chrono::DateTime::from_timestamp(c.expiration.secs(), c.expiration.subsec_nanos()), - }) - } - } else { - Ok(Credentials::Aws { - access_key_id: access_key.clone(), - secret_access_key: secret_key.clone(), - session_token: None, - expiration: None, - }) - } - } - } - } - - async fn presign_get(&self, _location: &str) -> Result { - // Stub: Presigning requires keeping an ObjectStore client around or rebuilding it. - // For this task, we focus on table credentials. - // Stub: Presigning requires keeping an ObjectStore client around or rebuilding it. - // For this task, we focus on table credentials. - Err(anyhow::anyhow!("MemoryStore does not support presigning yet")) - } - - -} - - -#[cfg(test)] -mod tests { - use super::*; - use pangolin_core::user::User; - use pangolin_core::permission::Permission; - use pangolin_core::model::{Tenant, Warehouse, AssetType}; - use std::collections::HashMap; - use chrono::Utc; - - #[tokio::test] - async fn test_tenant_operations() { - let store = MemoryStore::new(); - let tenant_id = Uuid::new_v4(); - let tenant = Tenant { - id: tenant_id, - name: "test_tenant".to_string(), - properties: HashMap::new(), - }; - - store.create_tenant(tenant.clone()).await.unwrap(); - let fetched = store.get_tenant(tenant_id).await.unwrap(); - assert!(fetched.is_some()); - assert_eq!(fetched.unwrap().name, "test_tenant"); - - let list = store.list_tenants().await.unwrap(); - assert_eq!(list.len(), 1); - } - - #[tokio::test] - async fn test_warehouse_operations() { - let store = MemoryStore::new(); - let tenant_id = Uuid::new_v4(); - let warehouse = Warehouse { - id: Uuid::new_v4(), - name: "main_warehouse".to_string(), - tenant_id, - storage_config: HashMap::new(), - use_sts: false, - vending_strategy: None, - }; - - store.create_warehouse(tenant_id, warehouse.clone()).await.unwrap(); - let fetched = store.get_warehouse(tenant_id, "main_warehouse".to_string()).await.unwrap(); - assert!(fetched.is_some()); - - let list = store.list_warehouses(tenant_id).await.unwrap(); - assert_eq!(list.len(), 1); - } - - #[tokio::test] - async fn test_asset_operations() { - let store = MemoryStore::new(); - let tenant_id = Uuid::new_v4(); - let catalog = "default"; - let namespace = vec!["ns1".to_string()]; - - let asset = Asset { - id: Uuid::new_v4(), - name: "tbl1".to_string(), - kind: AssetType::IcebergTable, - location: "s3://loc".to_string(), - properties: HashMap::new(), - }; - - store.create_asset(tenant_id, catalog, None, namespace.clone(), asset.clone()).await.unwrap(); - - let fetched = store.get_asset(tenant_id, catalog, None, namespace.clone(), "tbl1".to_string()).await.unwrap(); - assert!(fetched.is_some()); - - // Rename - store.rename_asset(tenant_id, catalog, None, namespace.clone(), "tbl1".to_string(), namespace.clone(), "tbl2".to_string()).await.unwrap(); - let old = store.get_asset(tenant_id, catalog, None, namespace.clone(), "tbl1".to_string()).await.unwrap(); - assert!(old.is_none()); - let new = store.get_asset(tenant_id, catalog, None, namespace.clone(), "tbl2".to_string()).await.unwrap(); - assert!(new.is_some()); - - store.delete_asset(tenant_id, catalog, None, namespace.clone(), "tbl2".to_string()).await.unwrap(); - let deleted = store.get_asset(tenant_id, catalog, None, namespace.clone(), "tbl2".to_string()).await.unwrap(); - assert!(deleted.is_none()); - } - - #[tokio::test] - async fn test_asset_update_consistency() { - let store = MemoryStore::new(); - crate::tests::test_asset_update_consistency(&store).await; - } - - #[tokio::test] - async fn test_list_permissions_filtering() { - use pangolin_core::permission::{Action, PermissionScope}; - use std::collections::HashSet; - - let store = MemoryStore::new(); - let tenant1 = Uuid::new_v4(); - let tenant2 = Uuid::new_v4(); - - // Create users for tenants - let user1 = Uuid::new_v4(); - let user2 = Uuid::new_v4(); - - let u1 = User { - id: user1, - username: "user1".to_string(), - email: "user1@example.com".to_string(), - password_hash: Some("hash".to_string()), - role: pangolin_core::user::UserRole::TenantUser, - tenant_id: Some(tenant1), - created_at: Utc::now(), - updated_at: Utc::now(), - last_login: None, - active: true, - oauth_provider: None, - oauth_subject: None, - }; - store.create_user(u1).await.unwrap(); - - let u2 = User { - id: user2, - username: "user2".to_string(), - email: "user2@example.com".to_string(), - password_hash: Some("hash".to_string()), - role: pangolin_core::user::UserRole::TenantUser, - tenant_id: Some(tenant2), - created_at: Utc::now(), - updated_at: Utc::now(), - last_login: None, - active: true, - oauth_provider: None, - oauth_subject: None, - }; - store.create_user(u2).await.unwrap(); - - // Grant permissions - let p1 = Permission::new( - user1, - PermissionScope::Catalog { catalog_id: Uuid::new_v4() }, - HashSet::from([Action::Read]), - Uuid::new_v4(), - ); - store.create_permission(p1.clone()).await.unwrap(); - - let p2 = Permission::new( - user2, - PermissionScope::Catalog { catalog_id: Uuid::new_v4() }, - HashSet::from([Action::Write]), - Uuid::new_v4(), - ); - store.create_permission(p2.clone()).await.unwrap(); - - // Test tenant filtering - let perms_t1 = store.list_permissions(tenant1).await.unwrap(); - assert_eq!(perms_t1.len(), 1); - assert_eq!(perms_t1[0].user_id, user1); - - let perms_t2 = store.list_permissions(tenant2).await.unwrap(); - assert_eq!(perms_t2.len(), 1); - assert_eq!(perms_t2[0].user_id, user2); - - // Test user filtering - let perms_u1 = store.list_user_permissions(user1).await.unwrap(); - assert_eq!(perms_u1.len(), 1); - assert_eq!(perms_u1[0].id, p1.id); - } - - #[tokio::test] - async fn test_list_user_permissions_aggregation() { - use pangolin_core::permission::{Action, PermissionScope, Role, UserRole}; - use std::collections::HashSet; - - let store = MemoryStore::new(); - let tenant_id = Uuid::new_v4(); - let user_id = Uuid::new_v4(); - let admin_id = Uuid::new_v4(); - - // 1. Create a Role with permissions - let mut role = Role::new("test-role".to_string(), None, tenant_id, admin_id); - let role_scope = PermissionScope::Catalog { catalog_id: Uuid::new_v4() }; - let mut role_actions = HashSet::new(); - role_actions.insert(Action::Read); - role.add_permission(role_scope.clone(), role_actions); - store.create_role(role.clone()).await.unwrap(); - - // 2. Assign role to user - let user_role = UserRole::new(user_id, role.id, admin_id); - store.assign_role(user_role).await.unwrap(); - - // 3. Create a direct permission - let direct_scope = PermissionScope::Catalog { catalog_id: Uuid::new_v4() }; - let mut direct_actions = HashSet::new(); - direct_actions.insert(Action::Write); - let direct_perm = Permission::new(user_id, direct_scope.clone(), direct_actions, admin_id); - store.create_permission(direct_perm.clone()).await.unwrap(); - - // 4. List user permissions and verify aggregation - let aggregated_perms = store.list_user_permissions(user_id).await.unwrap(); - - assert_eq!(aggregated_perms.len(), 2, "Should have 2 permissions (1 direct, 1 from role)"); - - let has_direct = aggregated_perms.iter().any(|p| p.scope == direct_scope && p.actions.contains(&Action::Write)); - let has_role_based = aggregated_perms.iter().any(|p| p.scope == role_scope && p.actions.contains(&Action::Read)); - - assert!(has_direct, "Aggregated permissions should include direct permission"); - assert!(has_role_based, "Aggregated permissions should include role-based permission"); - } -} - - diff --git a/pangolin/pangolin_store/src/mongo.rs.bak b/pangolin/pangolin_store/src/mongo.rs.bak deleted file mode 100644 index 4dd3df1..0000000 --- a/pangolin/pangolin_store/src/mongo.rs.bak +++ /dev/null @@ -1,2112 +0,0 @@ -use crate::CatalogStore; -use anyhow::Result; -use async_trait::async_trait; -use futures::stream::TryStreamExt; -use mongodb::{Client, Collection, Database}; -use mongodb::bson::{doc, Document, Bson, Binary}; -use mongodb::bson::spec::BinarySubtype; -use mongodb::options::{ClientOptions, ReplaceOptions}; -use pangolin_core::model::{ - Asset, AssetType, Branch, Catalog, Commit, Namespace, Tag, Tenant, Warehouse, VendingStrategy, - SystemSettings, SyncStats, -}; -use pangolin_core::user::{User, UserRole as CoreUserRole, OAuthProvider}; -use pangolin_core::permission::{Role, Permission, PermissionGrant, UserRole as UserRoleAssignment}; -use pangolin_core::business_metadata::{AccessRequest, RequestStatus, BusinessMetadata}; -use pangolin_core::token::TokenInfo; -use crate::signer::{Signer, Credentials}; -use object_store::ObjectStore; -use object_store::aws::AmazonS3Builder; -use pangolin_core::audit::AuditLogEntry; -use uuid::Uuid; -use std::collections::HashMap; -use chrono::Utc; -use std::sync::Arc; -use object_store::path::Path as ObjPath; - -#[derive(Clone)] -pub struct MongoStore { - client: Client, - db: Database, - object_store_cache: crate::ObjectStoreCache, - metadata_cache: crate::MetadataCache, -} - -impl MongoStore { - pub async fn new(connection_string: &str, database_name: &str) -> Result { - let mut client_options = ClientOptions::parse(connection_string).await?; - - // Configure connection pool from environment variables - if let Ok(max_pool_size) = std::env::var("MONGO_MAX_POOL_SIZE") { - if let Ok(size) = max_pool_size.parse::() { - client_options.max_pool_size = Some(size); - tracing::info!("MongoDB max pool size set to: {}", size); - } - } - - if let Ok(min_pool_size) = std::env::var("MONGO_MIN_POOL_SIZE") { - if let Ok(size) = min_pool_size.parse::() { - client_options.min_pool_size = Some(size); - tracing::info!("MongoDB min pool size set to: {}", size); - } - } - - // Set app name - client_options.app_name = Some("Pangolin".to_string()); - - let client = Client::with_options(client_options)?; - let db = client.database(database_name); - Ok(Self { - client, - db, - object_store_cache: crate::ObjectStoreCache::default(), - metadata_cache: crate::MetadataCache::default(), - }) - } - - fn tenants(&self) -> Collection { - self.db.collection("tenants") - } - - fn warehouses(&self) -> Collection { - self.db.collection("warehouses") - } - - fn catalogs(&self) -> Collection { - self.db.collection("catalogs") - } - - fn namespaces(&self) -> Collection { - self.db.collection("namespaces") - } - - fn assets(&self) -> Collection { - self.db.collection("assets") - } - - fn branches(&self) -> Collection { - self.db.collection("branches") - } - - fn tags(&self) -> Collection { - self.db.collection("tags") - } - - fn commits(&self) -> Collection { - self.db.collection("commits") - } - - fn audit_logs(&self) -> Collection { - self.db.collection("audit_logs") - } - - fn users(&self) -> Collection { - self.db.collection("users") - } - - fn roles(&self) -> Collection { - self.db.collection("roles") - } - - fn user_roles(&self) -> Collection { - self.db.collection("user_roles") - } - - fn permissions(&self) -> Collection { - self.db.collection("permissions") - } - - fn access_requests(&self) -> Collection { - self.db.collection("access_requests") - } - - fn business_metadata(&self) -> Collection { - self.db.collection("business_metadata") - } - - fn active_tokens(&self) -> Collection { - self.db.collection("active_tokens") - } - - fn system_settings(&self) -> Collection { - self.db.collection("system_settings") - } - - fn federated_sync_stats(&self) -> Collection { - self.db.collection("federated_sync_stats") - } - - fn merge_operations(&self) -> Collection { - self.db.collection("merge_operations") - } - - fn merge_conflicts(&self) -> Collection { - self.db.collection("merge_conflicts") - } - - fn service_users(&self) -> Collection { - self.db.collection("service_users") - } -} - -#[async_trait] -impl CatalogStore for MongoStore { - // Tenant Operations - async fn create_tenant(&self, tenant: Tenant) -> Result<()> { - self.tenants().insert_one(tenant).await?; - Ok(()) - } - - async fn get_tenant(&self, id: Uuid) -> Result> { - let filter = doc! { "id": to_bson_uuid(id) }; - let tenant = self.tenants().find_one(filter).await?; - Ok(tenant) - } - - async fn list_tenants(&self) -> Result> { - let cursor = self.tenants().find(doc! {}).await?; - let tenants: Vec = cursor.try_collect().await?; - Ok(tenants) - } - - async fn update_tenant(&self, tenant_id: Uuid, updates: pangolin_core::model::TenantUpdate) -> Result { - let filter = doc! { "id": to_bson_uuid(tenant_id) }; - let mut update_doc = doc! {}; - - if let Some(name) = updates.name { - update_doc.insert("name", name); - } - if let Some(properties) = updates.properties { - update_doc.insert("properties", bson::to_bson(&properties)?); - } - - if update_doc.is_empty() { - return self.get_tenant(tenant_id).await? - .ok_or_else(|| anyhow::anyhow!("Tenant not found")); - } - - let update = doc! { "$set": update_doc }; - self.tenants().update_one(filter.clone(), update).await?; - - self.get_tenant(tenant_id).await? - .ok_or_else(|| anyhow::anyhow!("Tenant not found")) - } - - async fn delete_tenant(&self, tenant_id: Uuid) -> Result<()> { - let filter = doc! { "id": to_bson_uuid(tenant_id) }; - let result = self.tenants().delete_one(filter).await?; - - if result.deleted_count == 0 { - return Err(anyhow::anyhow!("Tenant not found")); - } - Ok(()) - } - - // Warehouse Operations - async fn create_warehouse(&self, tenant_id: Uuid, warehouse: Warehouse) -> Result<()> { - // We might want to store tenant_id in the warehouse document if it's not already there - // The Warehouse struct has tenant_id. - self.warehouses().insert_one(warehouse).await?; - Ok(()) - } - - async fn get_warehouse(&self, tenant_id: Uuid, name: String) -> Result> { - let filter = doc! { "tenant_id": to_bson_uuid(tenant_id), "name": name }; - let warehouse = self.warehouses().find_one(filter).await?; - Ok(warehouse) - } - - async fn list_warehouses(&self, tenant_id: Uuid) -> Result> { - let filter = doc! { "tenant_id": to_bson_uuid(tenant_id) }; - let cursor = self.warehouses().find(filter).await?; - let warehouses: Vec = cursor.try_collect().await?; - Ok(warehouses) - } - - async fn update_warehouse(&self, tenant_id: Uuid, name: String, updates: pangolin_core::model::WarehouseUpdate) -> Result { - let filter = doc! { "tenant_id": to_bson_uuid(tenant_id), "name": &name }; - let mut update_doc = doc! {}; - - if let Some(new_name) = &updates.name { - update_doc.insert("name", new_name); - } - if let Some(config) = &updates.storage_config { - update_doc.insert("storage_config", bson::to_bson(config)?); - } - if let Some(use_sts) = updates.use_sts { - update_doc.insert("use_sts", use_sts); - } - if let Some(vending_strategy) = updates.vending_strategy { - update_doc.insert("vending_strategy", bson::to_bson(&vending_strategy)?); - } - - if update_doc.is_empty() { - return self.get_warehouse(tenant_id, name).await? - .ok_or_else(|| anyhow::anyhow!("Warehouse not found")); - } - - let update = doc! { "$set": update_doc }; - self.warehouses().update_one(filter, update).await?; - - let new_name = updates.name.unwrap_or(name); - self.get_warehouse(tenant_id, new_name).await? - .ok_or_else(|| anyhow::anyhow!("Warehouse not found")) - } - - async fn delete_warehouse(&self, tenant_id: Uuid, name: String) -> Result<()> { - let filter = doc! { "tenant_id": to_bson_uuid(tenant_id), "name": &name }; - let result = self.warehouses().delete_one(filter).await?; - - if result.deleted_count == 0 { - return Err(anyhow::anyhow!("Warehouse '{}' not found", name)); - } - Ok(()) - } - - // Catalog Operations - async fn create_catalog(&self, tenant_id: Uuid, catalog: Catalog) -> Result<()> { - // Catalog struct doesn't have tenant_id, so we need to wrap it or add it? - // Wait, Catalog struct in model.rs: - // pub struct Catalog { pub name: String, pub warehouse_name: Option, pub storage_location: Option, pub properties: HashMap } - // It doesn't have tenant_id. - // In Postgres we added a column. In Mongo we can wrap it in a document or just add the field dynamically if we use Document. - // But we are using typed Collection. - // We should probably use a wrapper struct for storage or just use Document. - // Let's use Document for flexibility here since we need to add tenant_id context. - - let mut doc = doc! { - "id": to_bson_uuid(catalog.id), - "tenant_id": to_bson_uuid(tenant_id), - "name": &catalog.name, - "catalog_type": format!("{:?}", catalog.catalog_type), - "properties": mongodb::bson::to_bson(&catalog.properties)? - }; - - // Add optional fields - if let Some(ref warehouse_name) = catalog.warehouse_name { - doc.insert("warehouse_name", warehouse_name); - } - if let Some(ref storage_location) = catalog.storage_location { - doc.insert("storage_location", storage_location); - } - if let Some(ref federated_config) = catalog.federated_config { - doc.insert("federated_config", mongodb::bson::to_bson(federated_config)?); - } - - self.db.collection::("catalogs").insert_one(doc).await?; - Ok(()) - } - - async fn get_catalog(&self, tenant_id: Uuid, name: String) -> Result> { - let filter = doc! { "tenant_id": to_bson_uuid(tenant_id), "name": name }; - let doc = self.db.collection::("catalogs").find_one(filter).await?; - Ok(doc) - } - - async fn list_catalogs(&self, tenant_id: Uuid) -> Result> { - let filter = doc! { "tenant_id": to_bson_uuid(tenant_id) }; - let cursor = self.db.collection::("catalogs").find(filter).await?; - let catalogs: Vec = cursor.try_collect().await?; - Ok(catalogs) - } - - async fn update_catalog(&self, tenant_id: Uuid, name: String, updates: pangolin_core::model::CatalogUpdate) -> Result { - let filter = doc! { "tenant_id": to_bson_uuid(tenant_id), "name": &name }; - let mut update_doc = doc! {}; - - if let Some(warehouse_name) = updates.warehouse_name { - update_doc.insert("warehouse_name", warehouse_name); - } - if let Some(storage_location) = updates.storage_location { - update_doc.insert("storage_location", storage_location); - } - if let Some(properties) = updates.properties { - update_doc.insert("properties", bson::to_bson(&properties)?); - } - - if update_doc.is_empty() { - return self.get_catalog(tenant_id, name).await? - .ok_or_else(|| anyhow::anyhow!("Catalog not found")); - } - - let update = doc! { "$set": update_doc }; - self.db.collection::("catalogs").update_one(filter, update).await?; - - self.get_catalog(tenant_id, name).await? - .ok_or_else(|| anyhow::anyhow!("Catalog not found")) - } - - async fn delete_catalog(&self, tenant_id: Uuid, name: String) -> Result<()> { - let filter = doc! { "tenant_id": to_bson_uuid(tenant_id), "name": &name }; // For catalog - // For children, filter is slightly different (catalog_name property) or similar - let child_filter = doc! { "tenant_id": to_bson_uuid(tenant_id), "catalog_name": &name }; - - // 1. Tags - self.db.collection::("tags").delete_many(child_filter.clone()).await?; - // 2. Branches - self.db.collection::("branches").delete_many(child_filter.clone()).await?; - // 3. Assets - self.db.collection::("assets").delete_many(child_filter.clone()).await?; - // 4. Namespaces - self.db.collection::("namespaces").delete_many(child_filter.clone()).await?; - - // 5. Catalog - let result = self.db.collection::("catalogs").delete_one(filter).await?; - - if result.deleted_count == 0 { - return Err(anyhow::anyhow!("Catalog '{}' not found", name)); - } - Ok(()) - } - - // Namespace Operations - async fn create_namespace(&self, tenant_id: Uuid, catalog_name: &str, namespace: Namespace) -> Result<()> { - let doc = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "name": &namespace.name, // Vec -> Array - "properties": mongodb::bson::to_bson(&namespace.properties)? - }; - self.db.collection::("namespaces").insert_one(doc).await?; - Ok(()) - } - - async fn get_namespace(&self, tenant_id: Uuid, catalog_name: &str, namespace: Vec) -> Result> { - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "name": namespace - }; - let doc = self.db.collection::("namespaces").find_one(filter).await?; - Ok(doc) - } - - async fn list_namespaces(&self, tenant_id: Uuid, catalog_name: &str, parent: Option) -> Result> { - let mut filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name - }; - - if let Some(p) = parent { - // This is tricky. Namespace name is Vec. - // "parent" usually implies a hierarchy. - // If parent is "a.b", we want "a.b.c", "a.b.d". - // We can filter where "name" starts with the parent components. - // But `parent` argument is String (dot joined?). The trait says `parent: Option`. - // In Postgres we did LIKE 'parent%'. - // Here we need to match array prefix. - // Let's assume parent string is dot-separated or something. - // Actually, `Namespace` struct has `name: Vec`. - // If parent is provided, we should convert it to Vec and check if it's a prefix. - // But `parent` is just a string. - // Let's assume for now we just list all and filter in memory or implement prefix match if possible. - // MVP: List all for catalog. - } - - let cursor = self.db.collection::("namespaces").find(filter).await?; - let namespaces: Vec = cursor.try_collect().await?; - Ok(namespaces) - } - - async fn delete_namespace(&self, tenant_id: Uuid, catalog_name: &str, namespace: Vec) -> Result<()> { - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "name": namespace - }; - self.db.collection::("namespaces").delete_one(filter).await?; - Ok(()) - } - - async fn update_namespace_properties(&self, tenant_id: Uuid, catalog_name: &str, namespace: Vec, properties: HashMap) -> Result<()> { - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "name": namespace - }; - - // We need to merge properties. - // $set: { "properties.key": "value" } - let mut set_doc = doc! {}; - for (k, v) in properties { - set_doc.insert(format!("properties.{}", k), v); - } - - let update = doc! { "$set": set_doc }; - self.db.collection::("namespaces").update_one(filter, update).await?; - Ok(()) - } - - // Asset Operations - async fn create_asset(&self, tenant_id: Uuid, catalog_name: &str, branch: Option, namespace: Vec, asset: Asset) -> Result<()> { - let branch_name = branch.unwrap_or_else(|| "main".to_string()); - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "branch": &branch_name, - "namespace": &namespace, - "name": &asset.name - }; - - let doc = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "branch": &branch_name, - "namespace": namespace, - "id": to_bson_uuid(asset.id), - "name": &asset.name, - "kind": format!("{:?}", asset.kind), - "location": &asset.location, - "properties": mongodb::bson::to_bson(&asset.properties)? - }; - - let options = ReplaceOptions::builder().upsert(true).build(); - self.db.collection::("assets").replace_one(filter, doc).with_options(options).await?; - Ok(()) - } - - async fn get_asset(&self, tenant_id: Uuid, catalog_name: &str, branch: Option, namespace: Vec, name: String) -> Result> { - let branch_name = branch.unwrap_or_else(|| "main".to_string()); - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "branch": branch_name, - "namespace": namespace, - "name": name - }; - // Note: Branch support in filter if needed, but usually asset name is unique in namespace? - // Or is it versioned by branch? - // In Postgres we had `active_branch`. - // Let's stick to the filter. - - let doc = self.db.collection::("assets").find_one(filter).await?; - - if let Some(d) = doc { - // Manual deserialization because we stored flattened fields - let kind_str = d.get_str("kind")?; - let kind = match kind_str { - "IcebergTable" => AssetType::IcebergTable, - "View" => AssetType::View, - _ => AssetType::IcebergTable, - }; - - let properties: HashMap = mongodb::bson::from_bson(d.get("properties").unwrap().clone())?; - - let id_bson = d.get("id").ok_or(anyhow::anyhow!("Missing id"))?; - let id = from_bson_uuid(id_bson)?; - - Ok(Some(Asset { - id, - name: d.get_str("name")?.to_string(), - kind, - location: d.get_str("location").unwrap_or("").to_string(), - properties, - })) - } else { - Ok(None) - } - } - - async fn list_assets(&self, tenant_id: Uuid, catalog_name: &str, branch: Option, namespace: Vec) -> Result> { - let branch_name = branch.unwrap_or_else(|| "main".to_string()); - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "branch": branch_name, - "namespace": namespace - }; - let cursor = self.db.collection::("assets").find(filter).await?; - let docs: Vec = cursor.try_collect().await?; - - let mut assets = Vec::new(); - for d in docs { - let kind_str = d.get_str("kind")?; - let kind = match kind_str { - "IcebergTable" => AssetType::IcebergTable, - "View" => AssetType::View, - _ => AssetType::IcebergTable, - }; - - let properties: HashMap = mongodb::bson::from_bson(d.get("properties").unwrap().clone())?; - let id = if let Ok(i) = d.get("id").ok_or(anyhow::anyhow!("Missing id")).and_then(|b| from_bson_uuid(b)) { - i - } else { - Uuid::new_v4() // Fallback if old data - }; - - assets.push(Asset { - id, - name: d.get_str("name")?.to_string(), - kind, - location: d.get_str("location").unwrap_or("").to_string(), - properties, - }); - } - Ok(assets) - } - - async fn get_asset_by_id(&self, tenant_id: Uuid, asset_id: Uuid) -> Result)>> { - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "id": to_bson_uuid(asset_id) - }; - let d = self.db.collection::("assets").find_one(filter).await?; - - if let Some(d) = d { - let catalog_name = d.get_str("catalog_name")?.to_string(); - let namespace = d.get_array("namespace")?.iter().map(|v| v.as_str().unwrap().to_string()).collect(); - let kind_str = d.get_str("kind")?; - let kind = match kind_str { - "IcebergTable" => AssetType::IcebergTable, - "View" => AssetType::View, - _ => AssetType::IcebergTable, - }; - - let properties: HashMap = mongodb::bson::from_bson(d.get("properties").unwrap().clone())?; - - let asset = Asset { - id: asset_id, - name: d.get_str("name")?.to_string(), - kind, - location: d.get_str("location").unwrap_or("").to_string(), - properties, - }; - - Ok(Some((asset, catalog_name, namespace))) - } else { - Ok(None) - } - } - - async fn delete_asset(&self, tenant_id: Uuid, catalog_name: &str, branch: Option, namespace: Vec, name: String) -> Result<()> { - let branch_name = branch.unwrap_or_else(|| "main".to_string()); - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "branch": branch_name, - "namespace": namespace, - "name": name - }; - self.db.collection::("assets").delete_one(filter).await?; - Ok(()) - } - - async fn rename_asset(&self, tenant_id: Uuid, catalog_name: &str, branch: Option, source_namespace: Vec, source_name: String, dest_namespace: Vec, dest_name: String) -> Result<()> { - let branch_name = branch.unwrap_or_else(|| "main".to_string()); - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "branch": &branch_name, - "namespace": source_namespace, - "name": source_name - }; - - let update = doc! { - "$set": { - "namespace": dest_namespace, - "name": dest_name - } - }; - - self.db.collection::("assets").update_one(filter, update).await?; - Ok(()) - } - - async fn count_namespaces(&self, tenant_id: Uuid) -> Result { - let filter = doc! { "tenant_id": to_bson_uuid(tenant_id) }; - let count = self.namespaces().count_documents(filter).await?; - Ok(count as usize) - } - - async fn count_assets(&self, tenant_id: Uuid) -> Result { - let filter = doc! { "tenant_id": to_bson_uuid(tenant_id) }; - let count = self.assets().count_documents(filter).await?; - Ok(count as usize) - } - - // Branch Operations - async fn create_branch(&self, tenant_id: Uuid, catalog_name: &str, branch: Branch) -> Result<()> { - let doc = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "name": &branch.name, - "head_commit_id": branch.head_commit_id, - "branch_type": format!("{:?}", branch.branch_type), - "assets": &branch.assets - }; - self.db.collection::("branches").insert_one(doc).await?; - Ok(()) - } - - async fn get_branch(&self, tenant_id: Uuid, catalog_name: &str, name: String) -> Result> { - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "name": name - }; - let doc = self.db.collection::("branches").find_one(filter).await?; - - if let Some(d) = doc { - let type_str = d.get_str("branch_type")?; - let branch_type = match type_str { - "Ingest" => pangolin_core::model::BranchType::Ingest, - "Experimental" => pangolin_core::model::BranchType::Experimental, - _ => pangolin_core::model::BranchType::Experimental, - }; - - Ok(Some(Branch { - name: d.get_str("name")?.to_string(), - head_commit_id: mongodb::bson::from_bson(d.get("head_commit_id").unwrap().clone())?, - branch_type, - assets: mongodb::bson::from_bson(d.get("assets").unwrap().clone())?, - })) - } else { - Ok(None) - } - } - - async fn list_branches(&self, tenant_id: Uuid, catalog_name: &str) -> Result> { - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name - }; - let cursor = self.db.collection::("branches").find(filter).await?; - let docs: Vec = cursor.try_collect().await?; - - let mut branches = Vec::new(); - for d in docs { - let type_str = d.get_str("branch_type")?; - let branch_type = match type_str { - "Ingest" => pangolin_core::model::BranchType::Ingest, - "Experimental" => pangolin_core::model::BranchType::Experimental, - _ => pangolin_core::model::BranchType::Experimental, - }; - - branches.push(Branch { - name: d.get_str("name")?.to_string(), - head_commit_id: mongodb::bson::from_bson(d.get("head_commit_id").unwrap().clone())?, - branch_type, - assets: mongodb::bson::from_bson(d.get("assets").unwrap().clone())?, - }); - } - Ok(branches) - } - - async fn merge_branch(&self, tenant_id: Uuid, catalog_name: &str, target_branch: String, source_branch: String) -> Result<()> { - let source = self.get_branch(tenant_id, catalog_name, source_branch.clone()).await? - .ok_or_else(|| anyhow::anyhow!("Source branch not found"))?; - - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "name": target_branch - }; - - let update = doc! { - "$set": { - "head_commit_id": source.head_commit_id - } - }; - - self.db.collection::("branches").update_one(filter, update).await?; - Ok(()) - } - - // Token Management - async fn list_active_tokens(&self, _tenant_id: Uuid, user_id: Uuid) -> Result> { - let filter = doc! { "user_id": to_bson_uuid(user_id) }; - let cursor = self.active_tokens().find(filter).await?; - let docs: Vec = cursor.try_collect().await?; - - let mut tokens = Vec::new(); - for d in docs { - tokens.push(TokenInfo { - id: from_bson_uuid(d.get("token_id").ok_or(anyhow::anyhow!("Missing token_id"))?)?, - tenant_id: Uuid::default(), // Not stored in mongo active_tokens - user_id: from_bson_uuid(d.get("user_id").ok_or(anyhow::anyhow!("Missing user_id"))?)?, - username: "unknown".to_string(), // Would need join - token: d.get_str("token").ok().map(|s| s.to_string()), - expires_at: mongodb::bson::from_bson(d.get("expires_at").unwrap().clone())?, - created_at: Utc::now(), // Not stored - is_valid: true, - }); - } - Ok(tokens) - } - - async fn store_token(&self, token_info: TokenInfo) -> Result<()> { - let doc = doc! { - "token_id": to_bson_uuid(token_info.id), - "user_id": to_bson_uuid(token_info.user_id), - "token": token_info.token.unwrap_or_default(), - "expires_at": token_info.expires_at - }; - self.active_tokens().insert_one(doc).await?; - Ok(()) - } - - // System Settings - async fn get_system_settings(&self, tenant_id: Uuid) -> Result { - let filter = doc! { "tenant_id": to_bson_uuid(tenant_id) }; - let doc = self.system_settings().find_one(filter).await?; - - if let Some(d) = doc { - Ok(mongodb::bson::from_bson(d.get("settings").unwrap().clone())?) - } else { - Ok(SystemSettings { - allow_public_signup: None, - default_warehouse_bucket: None, - default_retention_days: None, - smtp_host: None, - smtp_port: None, - smtp_user: None, - smtp_password: None, - }) - } - } - - async fn update_system_settings(&self, tenant_id: Uuid, settings: SystemSettings) -> Result { - let filter = doc! { "tenant_id": to_bson_uuid(tenant_id) }; - let update = doc! { - "$set": { - "settings": mongodb::bson::to_bson(&settings)? - } - }; - - let options = mongodb::options::UpdateOptions::builder().upsert(true).build(); - self.system_settings().update_one(filter, update).with_options(options).await?; - Ok(settings) - } - - // Service User Methods - async fn create_service_user(&self, service_user: pangolin_core::user::ServiceUser) -> Result<()> { - let doc = mongodb::bson::to_document(&service_user)?; - self.service_users().insert_one(doc).await?; - Ok(()) - } - - async fn get_service_user(&self, id: Uuid) -> Result> { - let filter = doc! { "_id": id.to_string() }; - if let Some(doc) = self.service_users().find_one(filter).await? { - Ok(Some(mongodb::bson::from_document(doc)?)) - } else { - Ok(None) - } - } - - async fn get_service_user_by_api_key_hash(&self, api_key_hash: &str) -> Result> { - let filter = doc! { "api_key_hash": api_key_hash }; - if let Some(doc) = self.service_users().find_one(filter).await? { - Ok(Some(mongodb::bson::from_document(doc)?)) - } else { - Ok(None) - } - } - - async fn list_service_users(&self, tenant_id: Uuid) -> Result> { - let filter = doc! { "tenant_id": tenant_id.to_string() }; - let mut cursor = self.service_users().find(filter).await?; - let mut users = Vec::new(); - - while cursor.advance().await? { - users.push(mongodb::bson::from_document(cursor.deserialize_current()?)?) -; - } - - Ok(users) - } - - async fn update_service_user( - &self, - id: Uuid, - name: Option, - description: Option, - active: Option, - ) -> Result<()> { - let filter = doc! { "_id": id.to_string() }; - let mut update_doc = doc! {}; - - if let Some(n) = name { - update_doc.insert("name", n); - } - if let Some(d) = description { - update_doc.insert("description", d); - } - if let Some(a) = active { - update_doc.insert("active", a); - } - - if !update_doc.is_empty() { - let update = doc! { "$set": update_doc }; - self.service_users().update_one(filter, update).await?; - } - - Ok(()) - } - - async fn delete_service_user(&self, id: Uuid) -> Result<()> { - let filter = doc! { "_id": id.to_string() }; - self.service_users().delete_one(filter).await?; - Ok(()) - } - - async fn update_service_user_last_used(&self, id: Uuid, timestamp: chrono::DateTime) -> Result<()> { - let filter = doc! { "_id": id.to_string() }; - let update = doc! { "$set": { "last_used": timestamp.to_rfc3339() } }; - self.service_users().update_one(filter, update).await?; - Ok(()) - } - - // Merge Operation Methods - async fn create_merge_operation(&self, operation: pangolin_core::model::MergeOperation) -> Result<()> { - let doc = mongodb::bson::to_document(&operation)?; - self.merge_operations().insert_one(doc).await?; - Ok(()) - } - - async fn get_merge_operation(&self, operation_id: Uuid) -> Result> { - let filter = doc! { "_id": operation_id.to_string() }; - if let Some(doc) = self.merge_operations().find_one(filter).await? { - Ok(Some(mongodb::bson::from_document(doc)?)) - } else { - Ok(None) - } - } - - async fn list_merge_operations(&self, tenant_id: Uuid, catalog_name: &str) -> Result> { - let filter = doc! { - "tenant_id": tenant_id.to_string(), - "catalog_name": catalog_name - }; - let mut cursor = self.merge_operations().find(filter).await?; - let mut operations = Vec::new(); - - while cursor.advance().await? { - operations.push(mongodb::bson::from_document(cursor.deserialize_current()?)?); - } - - Ok(operations) - } - - async fn update_merge_operation_status(&self, operation_id: Uuid, status: pangolin_core::model::MergeStatus) -> Result<()> { - let filter = doc! { "_id": operation_id.to_string() }; - let status_str = format!("{:?}", status); - let update = doc! { "$set": { "status": status_str } }; - self.merge_operations().update_one(filter, update).await?; - Ok(()) - } - - async fn complete_merge_operation(&self, operation_id: Uuid, result_commit_id: Uuid) -> Result<()> { - let filter = doc! { "_id": operation_id.to_string() }; - let update = doc! { - "$set": { - "status": "Completed", - "result_commit_id": result_commit_id.to_string(), - "completed_at": chrono::Utc::now().to_rfc3339() - } - }; - self.merge_operations().update_one(filter, update).await?; - Ok(()) - } - - async fn abort_merge_operation(&self, operation_id: Uuid) -> Result<()> { - let filter = doc! { "_id": operation_id.to_string() }; - let update = doc! { - "$set": { - "status": "Aborted", - "completed_at": chrono::Utc::now().to_rfc3339() - } - }; - self.merge_operations().update_one(filter, update).await?; - Ok(()) - } - - // Merge Conflict Methods - async fn create_merge_conflict(&self, conflict: pangolin_core::model::MergeConflict) -> Result<()> { - let doc = mongodb::bson::to_document(&conflict)?; - self.merge_conflicts().insert_one(doc).await?; - Ok(()) - } - - async fn get_merge_conflict(&self, conflict_id: Uuid) -> Result> { - let filter = doc! { "_id": conflict_id.to_string() }; - if let Some(doc) = self.merge_conflicts().find_one(filter).await? { - Ok(Some(mongodb::bson::from_document(doc)?)) - } else { - Ok(None) - } - } - - async fn list_merge_conflicts(&self, operation_id: Uuid) -> Result> { - let filter = doc! { "merge_operation_id": operation_id.to_string() }; - let mut cursor = self.merge_conflicts().find(filter).await?; - let mut conflicts = Vec::new(); - - while cursor.advance().await? { - conflicts.push(mongodb::bson::from_document(cursor.deserialize_current()?)?); - } - - Ok(conflicts) - } - - async fn resolve_merge_conflict(&self, conflict_id: Uuid, resolution: pangolin_core::model::ConflictResolution) -> Result<()> { - let filter = doc! { "_id": conflict_id.to_string() }; - let resolution_doc = mongodb::bson::to_document(&resolution)?; - let update = doc! { "$set": { "resolution": resolution_doc } }; - self.merge_conflicts().update_one(filter, update).await?; - Ok(()) - } - - async fn add_conflict_to_operation(&self, operation_id: Uuid, conflict_id: Uuid) -> Result<()> { - let filter = doc! { "_id": operation_id.to_string() }; - let update = doc! { "$addToSet": { "conflicts": conflict_id.to_string() } }; - self.merge_operations().update_one(filter, update).await?; - Ok(()) - } - - // Federated Catalog Operations - async fn sync_federated_catalog(&self, tenant_id: Uuid, catalog_name: &str) -> Result<()> { - let stats = SyncStats { - last_synced_at: Some(Utc::now()), - sync_status: "Success".to_string(), - tables_synced: 0, - namespaces_synced: 0, - error_message: None, - }; - - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name - }; - let update = doc! { - "$set": { - "stats": mongodb::bson::to_bson(&stats)? - } - }; - - let options = mongodb::options::UpdateOptions::builder().upsert(true).build(); - self.federated_sync_stats().update_one(filter, update).with_options(options).await?; - Ok(()) - } - - async fn get_federated_catalog_stats(&self, tenant_id: Uuid, catalog_name: &str) -> Result { - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name - }; - let doc = self.federated_sync_stats().find_one(filter).await?; - - if let Some(d) = doc { - Ok(mongodb::bson::from_bson(d.get("stats").unwrap().clone())?) - } else { - Ok(SyncStats { - last_synced_at: None, - sync_status: "Never Synced".to_string(), - tables_synced: 0, - namespaces_synced: 0, - error_message: None, - }) - } - } - - // Commit Operations - async fn create_commit(&self, tenant_id: Uuid, commit: Commit) -> Result<()> { - let mut doc = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "id": to_bson_uuid(commit.id), - "timestamp": commit.timestamp, - "author": &commit.author, - "message": &commit.message, - "operations": mongodb::bson::to_bson(&commit.operations)? - }; - if let Some(parent_id) = commit.parent_id { - doc.insert("parent_id", to_bson_uuid(parent_id)); - } else { - doc.insert("parent_id", Bson::Null); - } - self.db.collection::("commits").insert_one(doc).await?; - Ok(()) - } - - async fn get_commit(&self, tenant_id: Uuid, id: Uuid) -> Result> { - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "id": to_bson_uuid(id) - }; - let doc = self.db.collection::("commits").find_one(filter).await?; - - if let Some(d) = doc { - Ok(Some(Commit { - id: mongodb::bson::from_bson(d.get("id").unwrap().clone())?, - parent_id: mongodb::bson::from_bson(d.get("parent_id").unwrap().clone())?, - timestamp: d.get_i64("timestamp")?, - author: d.get_str("author")?.to_string(), - message: d.get_str("message")?.to_string(), - operations: mongodb::bson::from_bson(d.get("operations").unwrap().clone())?, - })) - } else { - Ok(None) - } - } - - // File Operations - - async fn read_file(&self, path: &str) -> Result> { - // Use metadata cache for metadata.json files - if path.ends_with("metadata.json") || path.ends_with(".metadata.json") { - return self.metadata_cache.get_or_fetch(path, || async { - self.read_file_uncached(path).await - }).await; - } - - // Non-metadata files bypass cache - self.read_file_uncached(path).await - } - - - async fn write_file(&self, path: &str, data: Vec) -> Result<()> { - // Invalidate metadata cache on write - if path.ends_with("metadata.json") || path.ends_with(".metadata.json") { - self.metadata_cache.invalidate(path).await; - } - - // Try to look up warehouse credentials first - if let Some(warehouse) = self.get_warehouse_for_location(path).await? { - if path.starts_with("s3://") || path.starts_with("az://") || path.starts_with("gs://") { - // Use cached object store - let cache_key = self.get_object_store_cache_key(&warehouse.storage_config, path); - let store = self.object_store_cache.get_or_insert(cache_key, || { - Arc::new(crate::object_store_factory::create_object_store(&warehouse.storage_config, path).unwrap()) - }); - - // Extract key relative to bucket - let key = if let Some(rest) = path.strip_prefix("s3://").or_else(|| path.strip_prefix("az://")).or_else(|| path.strip_prefix("gs://")) { - rest.split_once('/').map(|(_, k)| k).unwrap_or(rest) - } else { - path - }; - - store.put(&ObjPath::from(key), data.into()).await?; - return Ok(()); - } - } - - // Fallback to existing logic (Global Env Vars) - if let Some(rest) = path.strip_prefix("s3://") { - let (bucket, key) = rest.split_once('/').ok_or_else(|| anyhow::anyhow!("Invalid S3 path"))?; - - let mut builder = AmazonS3Builder::new() - .with_bucket_name(bucket) - .with_allow_http(true); - - if let Ok(endpoint) = std::env::var("S3_ENDPOINT") { - builder = builder.with_endpoint(endpoint); - } - if let Ok(key_id) = std::env::var("AWS_ACCESS_KEY_ID") { - builder = builder.with_access_key_id(key_id); - } - if let Ok(secret) = std::env::var("AWS_SECRET_ACCESS_KEY") { - builder = builder.with_secret_access_key(secret); - } - if let Ok(region) = std::env::var("AWS_REGION") { - builder = builder.with_region(region); - } - - let store = builder.build()?; - let location = ObjPath::from(key); - store.put(&location, data.into()).await?; - Ok(()) - } else { - Err(anyhow::anyhow!("Only s3:// paths are supported in Mongo store")) - } - } - - - // Tag Operations - async fn create_tag(&self, tenant_id: Uuid, catalog_name: &str, tag: Tag) -> Result<()> { - let doc = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "name": &tag.name, - "commit_id": to_bson_uuid(tag.commit_id) - }; - self.db.collection::("tags").insert_one(doc).await?; - Ok(()) - } - - async fn get_tag(&self, tenant_id: Uuid, catalog_name: &str, name: String) -> Result> { - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "name": name - }; - let doc = self.db.collection::("tags").find_one(filter).await?; - - if let Some(d) = doc { - Ok(Some(Tag { - name: d.get_str("name")?.to_string(), - commit_id: mongodb::bson::from_bson(d.get("commit_id").unwrap().clone())?, - })) - } else { - Ok(None) - } - } - - async fn list_tags(&self, tenant_id: Uuid, catalog_name: &str) -> Result> { - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name - }; - let cursor = self.db.collection::("tags").find(filter).await?; - let docs: Vec = cursor.try_collect().await?; - - let mut tags = Vec::new(); - for d in docs { - tags.push(Tag { - name: d.get_str("name")?.to_string(), - commit_id: mongodb::bson::from_bson(d.get("commit_id").unwrap().clone())?, - }); - } - Ok(tags) - } - - async fn delete_tag(&self, tenant_id: Uuid, catalog_name: &str, name: String) -> Result<()> { - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "name": name - }; - self.db.collection::("tags").delete_one(filter).await?; - Ok(()) - } - - // Audit Operations - async fn log_audit_event(&self, tenant_id: Uuid, event: AuditLogEntry) -> Result<()> { - let doc = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "id": to_bson_uuid(event.id), - "user_id": event.user_id.map(to_bson_uuid).unwrap_or(Bson::Null), - "username": &event.username, - "action": format!("{:?}", event.action), - "resource_type": format!("{:?}", event.resource_type), - "resource_id": event.resource_id.map(to_bson_uuid).unwrap_or(Bson::Null), - "resource_name": &event.resource_name, - "timestamp": mongodb::bson::DateTime::from_chrono(event.timestamp), - "ip_address": event.ip_address.as_ref().map(|s| s.as_str()).unwrap_or(""), - "user_agent": event.user_agent.as_ref().map(|s| s.as_str()).unwrap_or(""), - "result": format!("{:?}", event.result), - "error_message": event.error_message.as_ref().map(|s| s.as_str()).unwrap_or(""), - "metadata": mongodb::bson::to_bson(&event.metadata)? - }; - self.db.collection::("audit_logs").insert_one(doc).await?; - Ok(()) - } - - async fn list_audit_events(&self, tenant_id: Uuid, filter: Option) -> Result> { - let mut query = doc! { "tenant_id": to_bson_uuid(tenant_id) }; - - // Build filter conditions - if let Some(ref f) = filter { - if let Some(user_id) = f.user_id { - query.insert("user_id", to_bson_uuid(user_id)); - } - if let Some(ref action) = f.action { - query.insert("action", format!("{:?}", action)); - } - if let Some(ref resource_type) = f.resource_type { - query.insert("resource_type", format!("{:?}", resource_type)); - } - if let Some(resource_id) = f.resource_id { - query.insert("resource_id", to_bson_uuid(resource_id)); - } - if let Some(start_time) = f.start_time { - query.insert("timestamp", doc! { "$gte": mongodb::bson::DateTime::from_chrono(start_time) }); - } - if let Some(end_time) = f.end_time { - let existing = query.get_document_mut("timestamp").ok(); - if let Some(existing_doc) = existing { - existing_doc.insert("$lte", mongodb::bson::DateTime::from_chrono(end_time)); - } else { - query.insert("timestamp", doc! { "$lte": mongodb::bson::DateTime::from_chrono(end_time) }); - } - } - if let Some(ref result) = f.result { - query.insert("result", format!("{:?}", result)); - } - } - - // Build options with pagination - let limit = filter.as_ref().and_then(|f| f.limit).unwrap_or(100) as i64; - let skip = filter.as_ref().and_then(|f| f.offset).map(|o| o as u64); - - let mut options = mongodb::options::FindOptions::builder() - .sort(doc! { "timestamp": -1 }) - .limit(limit) - .build(); - - if let Some(skip_val) = skip { - options.skip = Some(skip_val); - } - - let cursor = self.db.collection::("audit_logs") - .find(query) - .with_options(options) - .await?; - let docs: Vec = cursor.try_collect().await?; - - let mut events = Vec::new(); - for d in docs { - // Parse action enum from string - let action_str = d.get_str("action")?; - let action = serde_json::from_str(&format!("\"{}\"" , action_str.to_lowercase())) - .unwrap_or(pangolin_core::audit::AuditAction::CreateCatalog); - - // Parse resource_type enum from string - let resource_type_str = d.get_str("resource_type")?; - let resource_type = serde_json::from_str(&format!("\"{}\"", resource_type_str.to_lowercase())) - .unwrap_or(pangolin_core::audit::ResourceType::Catalog); - - // Parse result enum from string - let result_str = d.get_str("result")?; - let result = serde_json::from_str(&format!("\"{}\"", result_str.to_lowercase())) - .unwrap_or(pangolin_core::audit::AuditResult::Success); - - events.push(AuditLogEntry { - id: mongodb::bson::from_bson(d.get("id").unwrap().clone())?, - tenant_id, - user_id: d.get("user_id").and_then(|b| from_bson_uuid(b).ok()), - username: d.get_str("username")?.to_string(), - action, - resource_type, - resource_id: d.get("resource_id").and_then(|b| from_bson_uuid(b).ok()), - resource_name: d.get_str("resource_name")?.to_string(), - timestamp: d.get_datetime("timestamp")?.to_chrono(), - ip_address: d.get_str("ip_address").ok().filter(|s| !s.is_empty()).map(|s| s.to_string()), - user_agent: d.get_str("user_agent").ok().filter(|s| !s.is_empty()).map(|s| s.to_string()), - result, - error_message: d.get_str("error_message").ok().filter(|s| !s.is_empty()).map(|s| s.to_string()), - metadata: mongodb::bson::from_bson(d.get("metadata").unwrap().clone())?, - }); - } - Ok(events) - } - - async fn get_audit_event(&self, tenant_id: Uuid, event_id: Uuid) -> Result> { - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "id": to_bson_uuid(event_id) - }; - - let doc = self.db.collection::("audit_logs").find_one(filter).await?; - - if let Some(d) = doc { - let action_str = d.get_str("action")?; - let action = serde_json::from_str(&format!("\"{}\"", action_str.to_lowercase())) - .unwrap_or(pangolin_core::audit::AuditAction::CreateCatalog); - - let resource_type_str = d.get_str("resource_type")?; - let resource_type = serde_json::from_str(&format!("\"{}\"", resource_type_str.to_lowercase())) - .unwrap_or(pangolin_core::audit::ResourceType::Catalog); - - let result_str = d.get_str("result")?; - let result = serde_json::from_str(&format!("\"{}\"", result_str.to_lowercase())) - .unwrap_or(pangolin_core::audit::AuditResult::Success); - - Ok(Some(AuditLogEntry { - id: mongodb::bson::from_bson(d.get("id").unwrap().clone())?, - tenant_id, - user_id: d.get("user_id").and_then(|b| from_bson_uuid(b).ok()), - username: d.get_str("username")?.to_string(), - action, - resource_type, - resource_id: d.get("resource_id").and_then(|b| from_bson_uuid(b).ok()), - resource_name: d.get_str("resource_name")?.to_string(), - timestamp: d.get_datetime("timestamp")?.to_chrono(), - ip_address: d.get_str("ip_address").ok().filter(|s| !s.is_empty()).map(|s| s.to_string()), - user_agent: d.get_str("user_agent").ok().filter(|s| !s.is_empty()).map(|s| s.to_string()), - result, - error_message: d.get_str("error_message").ok().filter(|s| !s.is_empty()).map(|s| s.to_string()), - metadata: mongodb::bson::from_bson(d.get("metadata").unwrap().clone())?, - })) - } else { - Ok(None) - } - } - - async fn count_audit_events(&self, tenant_id: Uuid, filter: Option) -> Result { - let mut query = doc! { "tenant_id": to_bson_uuid(tenant_id) }; - - // Build same filter conditions as list_audit_events - if let Some(ref f) = filter { - if let Some(user_id) = f.user_id { - query.insert("user_id", to_bson_uuid(user_id)); - } - if let Some(ref action) = f.action { - query.insert("action", format!("{:?}", action)); - } - if let Some(ref resource_type) = f.resource_type { - query.insert("resource_type", format!("{:?}", resource_type)); - } - if let Some(resource_id) = f.resource_id { - query.insert("resource_id", to_bson_uuid(resource_id)); - } - if let Some(start_time) = f.start_time { - query.insert("timestamp", doc! { "$gte": mongodb::bson::DateTime::from_chrono(start_time) }); - } - if let Some(end_time) = f.end_time { - let existing = query.get_document_mut("timestamp").ok(); - if let Some(existing_doc) = existing { - existing_doc.insert("$lte", mongodb::bson::DateTime::from_chrono(end_time)); - } else { - query.insert("timestamp", doc! { "$lte": mongodb::bson::DateTime::from_chrono(end_time) }); - } - } - if let Some(ref result) = f.result { - query.insert("result", format!("{:?}", result)); - } - } - - let count = self.db.collection::("audit_logs") - .count_documents(query) - .await? as usize; - - Ok(count) - } - - // User Operations - async fn create_user(&self, user: User) -> Result<()> { - self.users().insert_one(user).await?; - Ok(()) - } - - async fn get_user(&self, user_id: Uuid) -> Result> { - let filter = doc! { "id": to_bson_uuid(user_id) }; - let user = self.users().find_one(filter).await?; - Ok(user) - } - - async fn get_user_by_username(&self, username: &str) -> Result> { - let filter = doc! { "username": username }; - let user = self.users().find_one(filter).await?; - Ok(user) - } - - async fn list_users(&self, tenant_id: Option) -> Result> { - let filter = if let Some(tid) = tenant_id { - doc! { "tenant_id": to_bson_uuid(tid) } - } else { - doc! {} - }; - let cursor = self.users().find(filter).await?; - let users: Vec = cursor.try_collect().await?; - Ok(users) - } - - async fn update_user(&self, user: User) -> Result<()> { - let filter = doc! { "id": to_bson_uuid(user.id) }; - let mut doc = mongodb::bson::to_document(&user)?; - // Ensure UUIDs are stored as Binary Subtype 0 for consistency with filters - doc.insert("id", to_bson_uuid(user.id)); - if let Some(tid) = user.tenant_id { - doc.insert("tenant-id", to_bson_uuid(tid)); - } - let update = doc! { "$set": doc }; - self.db.collection::("users").update_one(filter, update).await?; - Ok(()) - } - - async fn delete_user(&self, user_id: Uuid) -> Result<()> { - let filter = doc! { "id": to_bson_uuid(user_id) }; - self.users().delete_one(filter).await?; - Ok(()) - } - - // Role Operations - async fn create_role(&self, role: Role) -> Result<()> { - self.roles().insert_one(role).await?; - Ok(()) - } - - async fn get_role(&self, role_id: Uuid) -> Result> { - let filter = doc! { "id": to_bson_uuid(role_id) }; - let role = self.roles().find_one(filter).await?; - Ok(role) - } - - async fn list_roles(&self, tenant_id: Uuid) -> Result> { - let filter = doc! { "tenant-id": to_bson_uuid(tenant_id) }; - let cursor = self.roles().find(filter).await?; - let roles: Vec = cursor.try_collect().await?; - Ok(roles) - } - - async fn delete_role(&self, role_id: Uuid) -> Result<()> { - let filter = doc! { "id": to_bson_uuid(role_id) }; - self.roles().delete_one(filter).await?; - Ok(()) - } - - async fn update_role(&self, role: Role) -> Result<()> { - let filter = doc! { "id": to_bson_uuid(role.id) }; - let mut doc = mongodb::bson::to_document(&role)?; - doc.insert("id", to_bson_uuid(role.id)); - doc.insert("tenant-id", to_bson_uuid(role.tenant_id)); - doc.insert("created-by", to_bson_uuid(role.created_by)); - - let update = doc! { "$set": doc }; - self.db.collection::("roles").update_one(filter, update).await?; - Ok(()) - } - - async fn assign_role(&self, user_role: UserRoleAssignment) -> Result<()> { - self.user_roles().insert_one(user_role).await?; - Ok(()) - } - - async fn revoke_role(&self, user_id: Uuid, role_id: Uuid) -> Result<()> { - let filter = doc! { - "user-id": to_bson_uuid(user_id), - "role-id": to_bson_uuid(role_id) - }; - self.user_roles().delete_one(filter).await?; - Ok(()) - } - - async fn get_user_roles(&self, user_id: Uuid) -> Result> { - let filter = doc! { "user-id": to_bson_uuid(user_id) }; - let cursor = self.user_roles().find(filter).await?; - let roles: Vec = cursor.try_collect().await?; - Ok(roles) - } - - // Permission Operations - async fn create_permission(&self, permission: Permission) -> Result<()> { - self.permissions().insert_one(permission).await?; - Ok(()) - } - - async fn revoke_permission(&self, permission_id: Uuid) -> Result<()> { - let filter = doc! { "id": to_bson_uuid(permission_id) }; - self.permissions().delete_one(filter).await?; - Ok(()) - } - - async fn list_user_permissions(&self, user_id: Uuid) -> Result> { - // 1. Fetch direct permissions - let filter = doc! { "user-id": to_bson_uuid(user_id) }; - let cursor = self.permissions().find(filter).await?; - let mut perms: Vec = cursor.try_collect().await?; - - // 2. Fetch role-based permissions - let user_roles = self.get_user_roles(user_id).await?; - for ur in user_roles { - if let Some(role) = self.get_role(ur.role_id).await? { - for grant in role.permissions { - perms.push(Permission { - id: Uuid::new_v4(), // Synthesized ID - user_id, - scope: grant.scope, - actions: grant.actions, - granted_by: role.created_by, - granted_at: role.created_at, - }); - } - } - } - - Ok(perms) - } - - async fn list_permissions(&self, tenant_id: Uuid) -> Result> { - // 1. Get all user IDs for the tenant - let user_filter = doc! { "tenant-id": to_bson_uuid(tenant_id) }; - let user_cursor = self.users().find(user_filter).await?; - let users: Vec = user_cursor.try_collect().await?; - let user_ids: Vec = users.iter().map(|u| to_bson_uuid(u.id)).collect(); - - if user_ids.is_empty() { - return Ok(vec![]); - } - - // 2. Get permissions for those users - let perm_filter = doc! { "user-id": { "$in": user_ids } }; - let perm_cursor = self.permissions().find(perm_filter).await?; - let perms: Vec = perm_cursor.try_collect().await?; - Ok(perms) - } - - // Maintenance Operations - async fn expire_snapshots(&self, _tenant_id: Uuid, _catalog_name: &str, _branch: Option, _namespace: Vec, _table: String, _retention_ms: i64) -> Result<()> { - Ok(()) - } - - // Business Metadata Operations - async fn upsert_business_metadata(&self, metadata: BusinessMetadata) -> Result<()> { - let filter = doc! { "asset-id": to_bson_uuid(metadata.asset_id) }; - let mut doc = mongodb::bson::to_document(&metadata)?; - doc.insert("asset-id", to_bson_uuid(metadata.asset_id)); - self.db.collection::("business_metadata") - .replace_one(filter, doc) - .upsert(true) - .await?; - Ok(()) - } - - async fn get_business_metadata(&self, asset_id: Uuid) -> Result> { - let filter = doc! { "asset-id": to_bson_uuid(asset_id) }; - let meta = self.business_metadata().find_one(filter).await?; - Ok(meta) - } - - async fn delete_business_metadata(&self, asset_id: Uuid) -> Result<()> { - let filter = doc! { "asset-id": to_bson_uuid(asset_id) }; - self.business_metadata().delete_one(filter).await?; - Ok(()) - } - - async fn search_assets(&self, tenant_id: Uuid, query: &str, tags: Option>) -> Result, String, Vec)>> { - let query_regex = mongodb::bson::Regex { - pattern: format!(".*{}.*", regex::escape(query)), - options: "i".to_string(), - }; - - let mut pipeline = vec![ - doc! { "$match": { "tenant_id": to_bson_uuid(tenant_id) } }, - doc! { - "$lookup": { - "from": "business_metadata", - "localField": "id", - "foreignField": "asset_id", - "as": "metadata" - } - }, - doc! { - "$unwind": { - "path": "$metadata", - "preserveNullAndEmptyArrays": true - } - }, - doc! { - "$match": { - "$or": [ - { "name": query_regex.clone() }, - { "metadata.description": query_regex } - ] - } - } - ]; - - if let Some(tag_list) = tags { - if !tag_list.is_empty() { - pipeline.push(doc! { - "$match": { - "metadata.tags": { "$all": tag_list } - } - }); - } - } - - let cursor = self.assets().aggregate(pipeline).await?; - let docs: Vec = cursor.try_collect().await?; - - let mut results = Vec::new(); - - for d in docs { - let metadata_doc = d.get_document("metadata").ok(); - - // Manual deserialization for Asset to ensure we get what we expect, - // though from_document works if struct matches. - // But we need catalog and namespace which are in the doc but not in the struct. - let asset: Asset = mongodb::bson::from_document(d.clone())?; - - let metadata = if let Some(md_doc) = metadata_doc { - if md_doc.is_empty() { None } else { Some(mongodb::bson::from_document(md_doc.clone())?) } - } else { None }; - - let catalog_name = d.get_str("catalog_name")?.to_string(); - let namespace_bson = d.get_array("namespace")?; - let namespace: Vec = namespace_bson.iter() - .map(|b| b.as_str().unwrap_or_default().to_string()) - .collect(); - - results.push((asset, metadata, catalog_name, namespace)); - } - - Ok(results) - } - - async fn search_catalogs(&self, tenant_id: Uuid, query: &str) -> Result> { - let query_regex = mongodb::bson::Regex { - pattern: format!(".*{}.*", regex::escape(query)), - options: "i".to_string(), - }; - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "name": query_regex - }; - let cursor = self.catalogs().find(filter).await?; - let catalogs: Vec = cursor.try_collect().await?; - Ok(catalogs) - } - - async fn search_namespaces(&self, tenant_id: Uuid, query: &str) -> Result> { - // Namespaces search with aggregation to output Docs - let query_regex = mongodb::bson::Regex { - pattern: format!(".*{}.*", regex::escape(query)), - options: "i".to_string(), - }; - - let pipeline = vec![ - doc! { "$match": { "tenant_id": to_bson_uuid(tenant_id) } } - ]; - - // self.namespaces() returns Collection. aggregate returns Cursor. - // BUT we need to call aggregate on collection. self.namespaces() is typed. - // We can call aggregate on typed collection but it returns Cursor. - let cursor = self.namespaces().aggregate(pipeline).await?; - let docs: Vec = cursor.try_collect().await?; - - let mut results = Vec::new(); - let query_lower = query.to_lowercase(); - - for d in docs { - let ns: Namespace = mongodb::bson::from_document(d.clone())?; - let catalog_name = d.get_str("catalog_name")?.to_string(); - - if ns.to_string().to_lowercase().contains(&query_lower) { - results.push((ns, catalog_name)); - } - } - Ok(results) - } - - async fn search_branches(&self, tenant_id: Uuid, query: &str) -> Result> { - // Use aggregate instead of find to get Document cursor easily and access catalog_name - let query_regex = mongodb::bson::Regex { - pattern: format!(".*{}.*", regex::escape(query)), - options: "i".to_string(), - }; - - // Explicitly filter by query in pipeline - let pipeline = vec![ - doc! { "$match": { "tenant_id": to_bson_uuid(tenant_id), "name": query_regex } } - ]; - - let cursor = self.branches().aggregate(pipeline).await?; - let docs: Vec = cursor.try_collect().await?; - - let mut results = Vec::new(); - for d in docs { - let branch: Branch = mongodb::bson::from_document(d.clone())?; - let catalog_name = d.get_str("catalog_name")?.to_string(); - results.push((branch, catalog_name)); - } - Ok(results) - } - - // Access Request Operations - async fn create_access_request(&self, request: AccessRequest) -> Result<()> { - self.access_requests().insert_one(request).await?; - Ok(()) - } - - async fn get_access_request(&self, id: Uuid) -> Result> { - let filter = doc! { "id": to_bson_uuid(id) }; - let req = self.access_requests().find_one(filter).await?; - Ok(req) - } - - async fn list_access_requests(&self, tenant_id: Uuid) -> Result> { - // AccessRequests stored with UserID/AssetID but not TenantID directly? - // Struct has: id, user_id, asset_id... - // User has tenant_id. - // To filter by tenant_id, we need a join (lookup) or we store tenant_id denormalized on AccessRequest? - // SQL implementation joins Users. - // Mongo: $lookup. - - // Creating aggregation pipeline: - let pipeline = vec![ - doc! { - "$lookup": { - "from": "users", - "localField": "user-id", - "foreignField": "id", - "as": "user" - } - }, - doc! { "$unwind": "$user" }, - doc! { "$match": { "user.tenant-id": to_bson_uuid(tenant_id) } }, - // Project back to AccessRequest root fields only? - // "replaceRoot"? Or simple map. - doc! { - "$project": { - "user": 0 // remove joined field to match struct - } - } - ]; - - let cursor = self.access_requests().aggregate(pipeline).await?; - // Cursor returns Documents, need to deserialize. - // aggregate returns Cursor. - let docs: Vec = cursor.try_collect().await?; - - let mut reqs = Vec::new(); - for d in docs { - reqs.push(mongodb::bson::from_document(d)?); - } - Ok(reqs) - } - - async fn update_access_request(&self, request: AccessRequest) -> Result<()> { - let filter = doc! { "id": to_bson_uuid(request.id) }; - self.access_requests().replace_one(filter, request).await?; - Ok(()) - } - - async fn remove_orphan_files(&self, _tenant_id: Uuid, _catalog_name: &str, _branch: Option, _namespace: Vec, _table: String, _older_than_ms: i64) -> Result<()> { - Ok(()) - } - - // Metadata IO - async fn get_metadata_location(&self, tenant_id: Uuid, catalog_name: &str, branch: Option, namespace: Vec, table: String) -> Result> { - let asset = self.get_asset(tenant_id, catalog_name, branch, namespace, table).await?; - if let Some(asset) = asset { - // First check if metadata_location is explicitly set in properties - if let Some(loc) = asset.properties.get("metadata_location") { - return Ok(Some(loc.clone())); - } - // Fall back to the asset's location field - return Ok(Some(asset.location)); - } - Ok(None) - } - - async fn update_metadata_location(&self, tenant_id: Uuid, catalog_name: &str, branch: Option, namespace: Vec, table: String, expected_location: Option, new_location: String) -> Result<()> { - let branch_name = branch.unwrap_or_else(|| "main".to_string()); - let filter = doc! { - "tenant_id": to_bson_uuid(tenant_id), - "catalog_name": catalog_name, - "branch": branch_name, - "namespace": namespace, - "name": table - }; - - // CAS Logic - need to check both location field and properties.metadata_location - // because get_metadata_location falls back to location if metadata_location doesn't exist - let mut query = filter.clone(); - if let Some(expected) = expected_location { - // Match if either properties.metadata_location equals expected OR location equals expected (and metadata_location doesn't exist) - query.insert("$or", vec![ - doc! { "properties.metadata_location": &expected }, - doc! { "location": &expected, "properties.metadata_location": doc! { "$exists": false } } - ]); - } else { - // expected is None, meaning it shouldn't exist or should be null. - query.insert("properties.metadata_location", doc! { "$exists": false }); - } - - let update = doc! { - "$set": { - "properties.metadata_location": new_location - } - }; - - let result = self.db.collection::("assets").update_one(query, update).await?; - - if result.matched_count == 0 { - return Err(anyhow::anyhow!("CAS check failed: Metadata location mismatch or asset not found")); - } - - Ok(()) - } - - // Token Revocation Operations - async fn revoke_token(&self, token_id: Uuid, expires_at: chrono::DateTime, reason: Option) -> Result<()> { - let revoked = pangolin_core::token::RevokedToken::new(token_id, expires_at, reason); - self.db.collection("revoked_tokens").insert_one(revoked).await?; - Ok(()) - } - - async fn is_token_revoked(&self, token_id: Uuid) -> Result { - let filter = doc! { "token_id": to_bson_uuid(token_id) }; - let result = self.db.collection::("revoked_tokens") - .find_one(filter) - .await?; - Ok(result.is_some()) - } - - async fn cleanup_expired_tokens(&self) -> Result { - let now = chrono::Utc::now(); - let filter = doc! { "expires_at": { "$lt": now } }; - let result = self.db.collection::("revoked_tokens") - .delete_many(filter) - .await?; - Ok(result.deleted_count as usize) - } -} - - -#[async_trait] -impl Signer for MongoStore { - async fn get_table_credentials(&self, location: &str) -> Result { - // Attempt to extract bucket/container from location - let (_scheme, container) = if location.starts_with("s3://") { - ("s3", location[5..].split('/').next().unwrap_or("").to_string()) - } else if location.starts_with("az://") { - ("az", location[5..].split('/').next().unwrap_or("").to_string()) - } else if location.starts_with("gs://") { - ("gs", location[5..].split('/').next().unwrap_or("").to_string()) - } else if location.starts_with("abfs://") { - ("abfs", location[7..].split('/').next().unwrap_or("").split('@').next().unwrap_or("").to_string()) - } else { - ("unknown", String::new()) - }; - - // Find warehouse matching this container - let filter = doc! { - "$or": [ - { "storage_config.s3.bucket": &container }, - { "storage_config.azure.container": &container }, - { "storage_config.gcp.bucket": &container } - ] - }; - - let warehouse = self.warehouses().find_one(filter).await? - .ok_or_else(|| anyhow::anyhow!("No warehouse found for location: {}", location))?; - - match &warehouse.vending_strategy { - Some(VendingStrategy::AwsSts { role_arn: _, external_id: _ }) => { - Err(anyhow::anyhow!("AWS STS vending not implemented yet via VendingStrategy in MongoStore")) - } - Some(VendingStrategy::AwsStatic { access_key_id, secret_access_key }) => { - Ok(Credentials::Aws { - access_key_id: access_key_id.clone(), - secret_access_key: secret_access_key.clone(), - session_token: None, - expiration: None, - }) - } - Some(VendingStrategy::AzureSas { account_name, account_key }) => { - let signer = crate::azure_signer::AzureSigner::new(account_name.clone(), account_key.clone()); - let sas_token = signer.generate_sas_token(location).await?; - Ok(Credentials::Azure { - sas_token, - account_name: account_name.clone(), - expiration: chrono::Utc::now() + chrono::Duration::hours(1), - }) - } - Some(VendingStrategy::GcpDownscoped { service_account_email, private_key }) => { - let signer = crate::gcp_signer::GcpSigner::new(service_account_email.clone(), private_key.clone()); - let access_token = signer.generate_downscoped_token(location).await?; - Ok(Credentials::Gcp { - access_token, - expiration: chrono::Utc::now() + chrono::Duration::hours(1), - }) - } - Some(VendingStrategy::None) => Err(anyhow::anyhow!("Vending disabled")), - None => { - // Backward compatibility logic - let access_key = warehouse.storage_config.get("s3.access-key-id") - .ok_or_else(|| anyhow::anyhow!("Missing s3.access-key-id"))?; - let secret_key = warehouse.storage_config.get("s3.secret-access-key") - .ok_or_else(|| anyhow::anyhow!("Missing s3.secret-access-key"))?; - - if warehouse.use_sts { - // Existing STS Logic restored for backward compatibility - let region = warehouse.storage_config.get("s3.region") - .map(|s| s.as_str()) - .unwrap_or("us-east-1"); - - let endpoint = warehouse.storage_config.get("s3.endpoint") - .map(|s| s.as_str()); - - let creds = aws_credential_types::Credentials::new( - access_key.to_string(), - secret_key.to_string(), - None, - None, - "legacy_provider" - ); - - let config_loader = aws_config::from_env() - .region(aws_config::Region::new(region.to_string())) - .credentials_provider(creds); - - let config = if let Some(ep) = endpoint { - config_loader.endpoint_url(ep).load().await - } else { - config_loader.load().await - }; - - let client = aws_sdk_sts::Client::new(&config); - - let role_arn = warehouse.storage_config.get("s3.role-arn").map(|s| s.as_str()); - - if let Some(arn) = role_arn { - let resp = client.assume_role() - .role_arn(arn) - .role_session_name("pangolin-mongo-legacy") - .send() - .await - .map_err(|e| anyhow::anyhow!("STS AssumeRole failed: {}", e))?; - - let c = resp.credentials.ok_or_else(|| anyhow::anyhow!("No credentials in AssumeRole response"))?; - Ok(Credentials::Aws { - access_key_id: c.access_key_id, - secret_access_key: c.secret_access_key, - session_token: Some(c.session_token), - expiration: chrono::DateTime::from_timestamp(c.expiration.secs(), c.expiration.subsec_nanos()), - }) - } else { - let resp = client.get_session_token() - .send() - .await - .map_err(|e| anyhow::anyhow!("STS GetSessionToken failed: {}", e))?; - - let c = resp.credentials.ok_or_else(|| anyhow::anyhow!("No credentials in GetSessionToken response"))?; - Ok(Credentials::Aws { - access_key_id: c.access_key_id, - secret_access_key: c.secret_access_key, - session_token: Some(c.session_token), - expiration: chrono::DateTime::from_timestamp(c.expiration.secs(), c.expiration.subsec_nanos()), - }) - } - } else { - Ok(Credentials::Aws { - access_key_id: access_key.clone(), - secret_access_key: secret_key.clone(), - session_token: None, - expiration: None, - }) - } - } - } - } - - async fn presign_get(&self, _location: &str) -> Result { - Err(anyhow::anyhow!("MongoStore does not support presigning yet")) - } -} - -fn to_bson_uuid(id: Uuid) -> Bson { - Bson::Binary(Binary { - subtype: BinarySubtype::Generic, - bytes: id.as_bytes().to_vec(), - }) -} - -fn from_bson_uuid(bson: &Bson) -> Result { - match bson { - Bson::Binary(Binary { subtype: BinarySubtype::Generic, bytes }) => { - Ok(Uuid::from_slice(bytes)?) - }, - _ => Err(anyhow::anyhow!("Invalid UUID bson")), - } -} - -impl MongoStore { - pub async fn create_user(&self, user: User) -> Result<()> { - self.users().insert_one(user).await?; - Ok(()) - } - - pub async fn get_user(&self, id: Uuid) -> Result> { - let filter = doc! { "id": to_bson_uuid(id) }; - let user = self.users().find_one(filter).await?; - Ok(user) - } - - pub async fn list_users(&self, tenant_id: Option) -> Result> { - let filter = if let Some(tid) = tenant_id { - doc! { "tenant-id": to_bson_uuid(tid) } - } else { - doc! {} - }; - let cursor = self.users().find(filter).await?; - let users: Vec = cursor.try_collect().await?; - Ok(users) - } - - async fn get_warehouse_for_location(&self, location: &str) -> Result> { - let cursor = self.warehouses().find(doc! {}).await.map_err(|e| anyhow::anyhow!(e))?; - let warehouses: Vec = cursor.try_collect().await.map_err(|e| anyhow::anyhow!(e))?; - - for warehouse in warehouses { - let s3_match = warehouse.storage_config.get("s3.bucket").or_else(|| warehouse.storage_config.get("bucket")).map(|b| location.contains(b)).unwrap_or(false); - let azure_match = warehouse.storage_config.get("azure.container").map(|c| location.contains(c)).unwrap_or(false); - let gcp_match = warehouse.storage_config.get("gcp.bucket").map(|b| location.contains(b)).unwrap_or(false); - - if s3_match || azure_match || gcp_match { - return Ok(Some(warehouse)); - } - } - - Ok(None) - } - - fn get_object_store_cache_key(&self, config: &HashMap, location: &str) -> String { - let endpoint = config.get("s3.endpoint").or_else(|| config.get("endpoint")).or_else(|| config.get("azure.endpoint")).or_else(|| config.get("gcp.endpoint")).map(|s| s.as_str()).unwrap_or(""); - let bucket = config.get("s3.bucket").or_else(|| config.get("bucket")).or_else(|| config.get("azure.container")).or_else(|| config.get("gcp.bucket")).map(|s| s.as_str()).unwrap_or_else(|| { - location.strip_prefix("s3://").or_else(|| location.strip_prefix("az://")).or_else(|| location.strip_prefix("gs://")).and_then(|s| s.split('/').next()).unwrap_or("") - }); - let access_key = config.get("s3.access-key-id").or_else(|| config.get("access_key_id")).or_else(|| config.get("azure.account-name")).or_else(|| config.get("gcp.service-account-key")).map(|s| s.as_str()).unwrap_or(""); - let region = config.get("s3.region").or_else(|| config.get("region")).or_else(|| config.get("azure.region")).or_else(|| config.get("gcp.region")).map(|s| s.as_str()).unwrap_or(""); - crate::ObjectStoreCache::cache_key(endpoint, &bucket, access_key, region) - } - - async fn read_file_uncached(&self, path: &str) -> Result> { - // Try to look up warehouse credentials first - if let Some(warehouse) = self.get_warehouse_for_location(path).await? { - if path.starts_with("s3://") || path.starts_with("az://") || path.starts_with("gs://") { - // Use cached object store - let cache_key = self.get_object_store_cache_key(&warehouse.storage_config, path); - let store = self.object_store_cache.get_or_insert(cache_key, || { - Arc::new(crate::object_store_factory::create_object_store(&warehouse.storage_config, path).unwrap()) - }); - - // Extract key relative to bucket - let key = if let Some(rest) = path.strip_prefix("s3://").or_else(|| path.strip_prefix("az://")).or_else(|| path.strip_prefix("gs://")) { - rest.split_once('/').map(|(_, k)| k).unwrap_or(rest) - } else { - path - }; - - match store.get(&ObjPath::from(key)).await { - Ok(result) => return Ok(result.bytes().await?.to_vec()), - Err(e) => { - tracing::warn!("Failed to read from warehouse-configured store for {}, falling back to global env: {}", path, e); - } - } - } - } - - if let Some(rest) = path.strip_prefix("s3://") { - let (bucket, key) = rest.split_once('/').ok_or_else(|| anyhow::anyhow!("Invalid S3 path"))?; - - let mut builder = AmazonS3Builder::new() - .with_bucket_name(bucket) - .with_allow_http(true); - - if let Ok(endpoint) = std::env::var("S3_ENDPOINT") { - builder = builder.with_endpoint(endpoint); - } - if let Ok(key_id) = std::env::var("AWS_ACCESS_KEY_ID") { - builder = builder.with_access_key_id(key_id); - } - if let Ok(secret) = std::env::var("AWS_SECRET_ACCESS_KEY") { - builder = builder.with_secret_access_key(secret); - } - if let Ok(region) = std::env::var("AWS_REGION") { - builder = builder.with_region(region); - } - - let store = builder.build()?; - let location = ObjPath::from(key); - let result = store.get(&location).await?; - let bytes = result.bytes().await?; - Ok(bytes.to_vec()) - } else { - Err(anyhow::anyhow!("Only s3:// paths are supported in Mongo store")) - } - } -} diff --git a/pangolin/scripts/check_env_var_docs.sh b/pangolin/scripts/check_env_var_docs.sh new file mode 100755 index 0000000..8d3ca49 --- /dev/null +++ b/pangolin/scripts/check_env_var_docs.sh @@ -0,0 +1,78 @@ +#!/usr/bin/env bash +# +# Verify the environment-variable reference against the source of truth. +# +# B43: `docs/environment-variables.md` documented `PANGOLIN_HOST`, +# `PANGOLIN_PORT` and `PANGOLIN_STORE_TYPE` - none of which any code reads - and +# omitted roughly twenty variables that are read. Anyone configuring a +# deployment from that page set variables that did nothing and missed the ones +# that mattered, which is exactly how B9's `PANGOLIN_STORE_TYPE` in the compose +# files survived. +# +# Hand-maintained documentation of a machine-readable fact drifts. This script +# re-derives the set from the code and fails when the docs and the code +# disagree, so the drift is caught in CI rather than by an auditor. +# +# Usage: scripts/check_env_var_docs.sh (run from the pangolin/ workspace root) + +set -euo pipefail + +DOC="../docs/environment-variables.md" + +if [[ ! -f "$DOC" ]]; then + echo "error: $DOC not found; run this from the pangolin/ workspace root" >&2 + exit 1 +fi + +# Every PANGOLIN_* name the server actually reads. Test-only knobs +# (PANGOLIN_TEST_*) are deliberately excluded: they configure the test harness, +# not a deployment. +mapfile -t IN_CODE < <( + grep -rhoE 'PANGOLIN_[A-Z0-9_]+' pangolin_api/src pangolin_store/src --include='*.rs' \ + | grep -v '^PANGOLIN_TEST_' \ + | sort -u +) + +# The doc deliberately names a few variables that do *not* exist, to warn +# readers off them. Those lines carry a `` marker so this +# check can tell "documented as real" from "documented as a trap". +mapfile -t IN_DOCS < <( + grep -vF '' "$DOC" \ + | grep -ohE 'PANGOLIN_[A-Z0-9_]+' \ + | grep -v '^PANGOLIN_TEST_' \ + | sort -u +) + +missing=() +for name in "${IN_CODE[@]}"; do + if ! printf '%s\n' "${IN_DOCS[@]}" | grep -qx "$name"; then + missing+=("$name") + fi +done + +phantom=() +for name in "${IN_DOCS[@]}"; do + if ! printf '%s\n' "${IN_CODE[@]}" | grep -qx "$name"; then + phantom+=("$name") + fi +done + +status=0 + +if (( ${#missing[@]} )); then + echo "error: read by the server but absent from $DOC:" >&2 + printf ' %s\n' "${missing[@]}" >&2 + status=1 +fi + +if (( ${#phantom[@]} )); then + echo "error: documented in $DOC but read by nothing:" >&2 + printf ' %s\n' "${phantom[@]}" >&2 + status=1 +fi + +if (( status == 0 )); then + echo "environment-variable reference matches the code (${#IN_CODE[@]} variables)" +fi + +exit "$status" diff --git a/pangolin_ui/.gitignore b/pangolin_ui/.gitignore index 3b462cb..71f58cb 100644 --- a/pangolin_ui/.gitignore +++ b/pangolin_ui/.gitignore @@ -21,3 +21,12 @@ Thumbs.db # Vite vite.config.js.timestamp-* vite.config.ts.timestamp-* + +# Test and debug output (B45). +# +# ~260 KB of `check_*.txt` scratch files and Playwright artefacts were tracked +# in git. The ignore patterns did not cover them, and by the time they were +# added the files were already tracked - so `.gitignore` had no effect on them. +test-results/ +playwright-report/ +check_*.txt diff --git a/pangolin_ui/check_catalogs_list.txt b/pangolin_ui/check_catalogs_list.txt deleted file mode 100644 index dd960f2..0000000 --- a/pangolin_ui/check_catalogs_list.txt +++ /dev/null @@ -1,971 +0,0 @@ - -> pangolin-ui@0.1.0 check -> svelte-kit sync && svelte-check --tsconfig ./tsconfig.json - -Loading svelte-check in workspace: /home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui -Getting Svelte diagnostics... - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/lib/components/ui/Modal.svelte:30:2 -Warn: Elements with the 'dialog' interactive role must have a tabindex value -https://svelte.dev/e/a11y_interactive_supports_focus (svelte) -{#if open} -
e.key === 'Escape' && close()} - role="dialog" - aria-modal="true" - > -
- -
- - -
- - {#if title} -
-

{title}

-
- {/if} - - -
- -
- - -
- - - -
-
-
-
-{/if} - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/lib/components/ui/DataTable.svelte:105:9 -Warn: Self-closing HTML tags for non-void elements are ambiguous — use `
` rather than `
` -https://svelte.dev/e/element_invalid_self_closing_tag (svelte) -
-
- Loading... - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/lib/components/ui/Textarea.svelte:18:5 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) - {#if label} - - {/if} - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/branches/+page.svelte:97:5 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- - - - ` rather than `` rather than ` - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/assets/[id]/+page.svelte:194:33 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- - - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/assets/[id]/+page.svelte:204:33 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- - - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/branches/[catalog]/[name]/+page.svelte:114:5 -Warn: Self-closing HTML tags for non-void elements are ambiguous — use `
` rather than `
` -https://svelte.dev/e/element_invalid_self_closing_tag (svelte) -
-
-
- -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/catalogs/[name]/+page.svelte:108:5 -Warn: Self-closing HTML tags for non-void elements are ambiguous — use `
` rather than `
` -https://svelte.dev/e/element_invalid_self_closing_tag (svelte) -
-
-
- -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/catalogs/[name]/+page.svelte:193:7 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- - ` rather than `` rather than ` - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/assets/[id]/+page.svelte:194:33 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- - - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/assets/[id]/+page.svelte:204:33 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- - - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/branches/[catalog]/[name]/+page.svelte:114:5 -Warn: Self-closing HTML tags for non-void elements are ambiguous — use `
` rather than `
` -https://svelte.dev/e/element_invalid_self_closing_tag (svelte) -
-
-
- -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/catalogs/[name]/+page.svelte:108:5 -Warn: Self-closing HTML tags for non-void elements are ambiguous — use `
` rather than `
` -https://svelte.dev/e/element_invalid_self_closing_tag (svelte) -
-
-
- -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/catalogs/[name]/+page.svelte:193:7 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- - ` rather than `` rather than ` - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/assets/[id]/+page.svelte:194:33 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- - - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/assets/[id]/+page.svelte:204:33 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- - - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/branches/[catalog]/[name]/+page.svelte:114:5 -Warn: Self-closing HTML tags for non-void elements are ambiguous — use `
` rather than `
` -https://svelte.dev/e/element_invalid_self_closing_tag (svelte) -
-
-
- -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/catalogs/[name]/+page.svelte:108:5 -Warn: Self-closing HTML tags for non-void elements are ambiguous — use `
` rather than `
` -https://svelte.dev/e/element_invalid_self_closing_tag (svelte) -
-
-
- -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/catalogs/[name]/+page.svelte:193:7 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- - ` rather than `` rather than ` - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/assets/[id]/+page.svelte:194:33 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- - - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/assets/[id]/+page.svelte:204:33 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- - - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/branches/[catalog]/[name]/+page.svelte:114:5 -Warn: Self-closing HTML tags for non-void elements are ambiguous — use `
` rather than `
` -https://svelte.dev/e/element_invalid_self_closing_tag (svelte) -
-
-
- -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/catalogs/[name]/+page.svelte:108:5 -Warn: Self-closing HTML tags for non-void elements are ambiguous — use `
` rather than `
` -https://svelte.dev/e/element_invalid_self_closing_tag (svelte) -
-
-
- -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/catalogs/[name]/+page.svelte:193:7 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- - ` rather than `` rather than ` - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/assets/[id]/+page.svelte:194:33 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- - - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/assets/[id]/+page.svelte:204:33 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- - - -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/branches/[catalog]/[name]/+page.svelte:114:5 -Warn: Self-closing HTML tags for non-void elements are ambiguous — use `
` rather than `
` -https://svelte.dev/e/element_invalid_self_closing_tag (svelte) -
-
-
- -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/catalogs/[name]/+page.svelte:108:5 -Warn: Self-closing HTML tags for non-void elements are ambiguous — use `
` rather than `
` -https://svelte.dev/e/element_invalid_self_closing_tag (svelte) -
-
-
- -/home/alexmerced/development/personal/Personal/2026/pangolin/pangolin_ui/src/routes/catalogs/[name]/+page.svelte:193:7 -Warn: A form label must be associated with a control -https://svelte.dev/e/a11y_label_has_associated_control (svelte) -
- -