Implement roadmap_aug10: authorization, storage parity, Iceberg conformance and the guardrails that catch them

.github/workflows/build-binaries.yml

@@ -29,7 +29,19 @@ jobs:
           path: deployment_assets/bin/linux-x86_64/*
 
   build-macos-intel:
-    runs-on: macos-13  # Intel runner
+    # `macos-15-intel`, not `macos-13`.
+    #
+    # The macOS 13 image was retired in December 2025. Every tagged release
+    # since has hung here: the job sat queued for the full 24-hour limit
+    # waiting for a runner that no longer exists, while linux, macos-arm and
+    # windows finished in 8-15 minutes, and `create-release` was skipped
+    # because it `needs:` all four. That is why v0.4.0 through v0.6.0 produced
+    # no GitHub release and no binaries - the workflow had never once
+    # succeeded.
+    #
+    # `macos-15-intel` is GitHub's documented replacement for workflows that
+    # genuinely need x86_64.
+    runs-on: macos-15-intel
     steps:
       - uses: actions/checkout@v4
       - uses: dtolnay/rust-toolchain@stable
@@ -98,8 +110,13 @@ jobs:
     steps:
       - uses: actions/download-artifact@v4
       - name: Create Release
-        uses: softprops/action-gh-release@v1
+        uses: softprops/action-gh-release@v2
         with:
+          body: |
+            See [CHANGELOG.md](https://github.com/AlexMercedCoder/Pangolin/blob/main/CHANGELOG.md)
+            for the full list of changes, and
+            [SECURITY.md](https://github.com/AlexMercedCoder/Pangolin/blob/main/SECURITY.md)
+            for the security advisory and upgrade steps.
           files: |
             linux-binaries/*
             macos-intel-binaries/*

.gitignore

@@ -53,3 +53,14 @@ check_log.txt
 test_results.txt
 debug_output*.txt
 :memory:
+
+# Editor backup copies of source files (B45).
+#
+# `pangolin_store/src/memory.rs.bak` and `mongo.rs.bak` sat in the tree as ~4k
+# lines of divergent, dead query copies - a grep trap that returned plausible
+# but stale code for anyone searching the store layer.
+*.bak
+*.rs.bak
+
+# Logs live under logs/ directories that were previously tracked.
+logs/

CONTRIBUTING.md

@@ -5,7 +5,7 @@ recently, nothing at all for contributors — this file closes that gap.
 
 ## From clone to green
 
-You need Rust 1.92 or newer and Docker (only for the database-backed tests).
+You need Rust 1.94 or newer and Docker (only for the database-backed tests).
 
 ```bash
 git clone https://github.com/AlexMercedCoder/pangolin

README.md

@@ -25,7 +25,7 @@ Finally, pangolins are rare and specialized. They exist for a specific purpose a
 ## 🚀 Quick Start
 
 ### Prerequisites
-- Rust 1.92+
+- Rust 1.94+
 - Docker (optional, for MinIO)
 
 ### Running Locally

SECURITY.md

@@ -41,8 +41,130 @@ security fixes.
 
 | Version | Supported |
 |---|---|
-| 0.6.x | Yes |
-| 0.5.x and earlier | No — upgrade to 0.6.x |
+| 0.7.x | Yes |
+| 0.6.x | No — upgrade to 0.7.x, see the advisory below |
+| 0.5.x and earlier | No — upgrade to 0.7.x |
+
+## Fixed in 0.7.0
+
+**Every version before 0.7.0 is affected by a privilege-escalation
+vulnerability and should not be run.** Upgrade to 0.7.0 and rotate every issued
+token. The findings below come from a full-repo audit conducted the day after
+the 0.6.0 release; the authorization cluster was outside that release's scope
+and is new information, not a regression.
+
+Note which version you are actually running. The published container image
+`alexmerced/pangolin-api` was last pushed at **0.5.1** — no 0.6.0 image was
+ever released — so most deployments are on 0.5.1, which carries everything
+described here *and* everything 0.6.0 fixed. The affected range is
+`< 0.7.0`, not `0.6.0` alone.
+
+### Exploitable by any authenticated principal
+
+Every issue in this table needs nothing but a valid credential — including the
+lowest-privilege `tenant-user` account, or any service-user API key.
+
+| ID | Issue | Impact |
+|---|---|---|
+| B0a | `POST /api/v1/tokens` took no session and mapped a body-supplied `roles: ["Root"]` straight into signed claims | **Full privilege escalation.** Any authenticated caller could mint a valid `Root` token for any tenant. `check_permission` short-circuits for `Root`, so the resulting token bypasses every subsequent authorization check in the system |
+| B0b | The credential-vending endpoint performed no authorization, never resolved the table, and hardcoded `["read", "write"]` | **Cloud storage credential disclosure.** Any tenant member obtained read+write credentials for the entire warehouse, naming a table they had no rights to and which need not exist |
+| B0j | Logout revoked `session.user_id`; revocation is keyed by the token's `jti`, which no token carries as its `user_id` | **Logout did nothing.** The token stayed valid for its full 24-hour lifetime. On a shared machine, "signing out" left a working credential behind |
+| B0g | The Iceberg OAuth token endpoint checked `active` but not expiry, unlike the API-key path | **Expired credentials were renewable indefinitely.** An expired service user could exchange `client_credentials` for a fresh JWT, repeatedly |
+| B0h | The `NO_AUTH` public-bind guard read `no_auth && !dev_mode && !is_loopback(..)` | **Unauthenticated tenant-admin access.** `PANGOLIN_NO_AUTH=true` with `PANGOLIN_DEV_MODE=true` — routinely set together in compose and development setups — started happily on `0.0.0.0` and treated every anonymous request as `TenantAdmin` |
+| B0l | OAuth matched existing users on email with no `email_verified` check and no provider binding | **Account takeover.** Anyone able to set a matching address on any configured provider — GitHub permits unverified addresses — logged in as that Pangolin user, including the seeded tenant admin |
+| B0i | `PermissionScope::Tenant` matched without comparing the grant's tenant to the resource's | A tenant-wide grant issued in one tenant satisfied authorization for another tenant's resources |
+| B0c–B0f | `rename_table`, `update_namespace_properties`, view create/read and `perform_maintenance` performed no authorization check at all | Any tenant member could move any table (an effective delete), rewrite namespace properties including `location`, read any view's SQL, and trigger snapshot expiry and orphan-file deletion. `perform_maintenance` additionally ran against a hardcoded `"default"` catalog rather than the one addressed |
+| B0m | `expires_in_hours` reached `chrono::Duration::hours` unclamped, and no catch-panic layer was installed | **Remote panic.** A large value aborted the connection task, taking other in-flight requests on that connection with it |
+| B0o | A token whose `jti` was absent or unparseable skipped the revocation check entirely | Such tokens were unrevocable for their full lifetime |
+
+### Availability
+
+| ID | Issue | Impact |
+|---|---|---|
+| — | `SqliteStore` had no inherent `revoke_token`/`is_token_revoked`, so the trait implementations called themselves | **Remote crash on the SQLite backend.** Revoking a token — which logout does — recursed until the thread stack was exhausted and aborted the process |
+| B2 | On MongoDB the revocation write and the revocation check used different field names *and* different types | Revocation could never match. Revoked tokens, including after logout, stayed valid |
+
+### Confidentiality and integrity of records
+
+| ID | Issue | Impact |
+|---|---|---|
+| B1 | MongoDB's `get_audit_event` discarded the `tenant_id` parameter | **Cross-tenant audit disclosure.** Any tenant holding an audit-event UUID could read another tenant's record: username, IP address, resource names and metadata |
+| — | The SQLite `audit_logs` table declared different columns than the code inserted | **No audit trail at all on SQLite.** Every audit write failed at runtime. If you run SQLite, assume you have no audit history prior to 0.7.0 |
+| — | The admin CLI and Python SDK wrote their stored bearer tokens at the process umask, typically `0644` | Any local account could read the token. Now `0600` under a `0700` directory |
+| — | `generate-code` and `get-token` printed live JWTs into copy-paste output | Tokens landed in shell scrollback, session transcripts and pasted snippets |
+| B44 | A live PyPI API publish token was committed to the repository working tree in `.env` | Never tracked by Git, but readable by any local tool and passed into containers by `docker compose`. **The token has been removed and must be treated as compromised** |
+
+### Data loss
+
+Not security boundaries, but silent corruption is worth the same attention:
+
+* **B5** MongoDB's `update_metadata_location` ignored the compare-and-swap
+  entirely. Two concurrent Iceberg commits both reported success and one
+  snapshot was lost. Memory, Postgres and SQLite all enforced it.
+* **B3** SQLite's `delete_branch` referenced a column that does not exist and
+  was not transactional: the branch was committed away and its assets orphaned
+  permanently, while the caller received an error suggesting nothing happened.
+* **B7** All three persistent backends stored the `Debug` spelling of
+  `AssetType` and parsed only two of seventeen variants. A `DeltaTable`,
+  `MlModel` or `Lance` asset round-tripped as an Iceberg table.
+* **B4** Postgres decoded a `TEXT[]` column as `String`; `sqlx::Row::get`
+  panics on a decode failure, so any search returning at least one hit panicked
+  the request.
+
+### Credential vending could not be built
+
+The `aws-sts`, `azure-oauth` and `gcp-oauth` features - and the
+`cloud-credentials` bundle that unions them - did not compile, at any version.
+`cargo build` and `cargo test` run with default features and no job ever passed
+`--features`, so the entire cloud-credential surface had rotted into code that
+could not be built at all: parameters bound as `_name` and referenced as `name`
+inside the `cfg` block, a missing macro import, and an STS expiry parsed from a
+`DateTime` as though it were an RFC3339 string.
+
+This is not an exploitable defect - unbuildable code ships in no binary. It
+matters because it means **STS-based credential vending was not running
+anywhere**, so any deployment that believed it was handing out scoped,
+time-limited credentials was not. Check what your warehouses are actually
+configured with: if `use_sts` is set but the server was built without the
+feature (which is to say, always), the static-credential fallback in
+`S3Signer::generate_credentials` is what answered - vending your long-lived
+warehouse keys, with no expiry, instead of a scoped session token. Where no
+static keys were configured either, the call failed with "AWS credentials not
+configured", which at least failed closed.
+
+Fixed in 0.7.0, with a CI job that builds every optional feature so it cannot
+recur silently.
+
+### Upgrading to 0.7.0
+
+1. **Rotate every issued token.** B0a means any account may have minted a
+   `Root` token, and B0j means logging out never invalidated anything. Rotating
+   `PANGOLIN_JWT_SECRET` invalidates all existing sessions at once and is the
+   fastest way to be sure.
+2. **Rotate service-user API keys**, for the same reason: B0g allowed expired
+   keys to keep issuing fresh JWTs.
+3. **Rotate any cloud storage credentials** reachable through a warehouse that
+   an untrusted tenant member could name. B0b vended them to anyone.
+4. **Audit for the escalation.** `POST /api/v1/tokens` calls from non-admin
+   principals, and credential-vending calls for tables the caller had no grant
+   on, are the two signals. Note that on SQLite there is no audit history to
+   check, and on MongoDB action-filtered queries returned nothing (B23) — so a
+   clean audit log is not evidence of absence on those backends.
+5. **Check `PANGOLIN_NO_AUTH` and `PANGOLIN_DEV_MODE`.** If both were set on a
+   non-loopback bind, treat the deployment as having been open to anonymous
+   tenant-admin access for that period. The server now refuses to start in that
+   configuration.
+6. **Set `PANGOLIN_OAUTH_EMAIL_LINK_DOMAINS`** if you rely on OAuth accounts
+   being matched to existing local users by email. Without it, identity is
+   `(provider, subject)` only and an address never adopts an existing account —
+   which is the safe default, and a behaviour change.
+7. **On SQLite, upgrading migrates the `audit_logs` table** to schema version 2.
+   The previous table is preserved as `audit_logs_pre_v2`; it is empty in
+   practice, because nothing could ever write to it.
+8. **Third-party API clients may need changes.** Server request bodies now
+   reject unknown fields with `422` instead of silently ignoring them. If a
+   client sent a misspelled or obsolete field, it was already being discarded —
+   the request was never doing what it appeared to.
 
 ## Fixed in 0.6.0
 
@@ -126,5 +248,12 @@ Stated plainly, because a checklist that hides its limits is worse than none:
 * **Audit records are not tamper-evident.** They live in the same database as
   application data, with no hash chaining and no WORM option.
 * **No MFA, password policy, or account lockout.**
+* **Eight dependency advisories are accepted rather than fixed**, each with its
+  reasoning recorded in `.cargo/audit.toml`. None is reachable in a default
+  build, but they are exceptions rather than absences: `rsa` (never compiled),
+  `quick-xml` (held back by `object_store` 0.11 and `azure_core` 0.20),
+  a second `rustls-webpki` arriving via the AWS SDK's `rustls` 0.21,
+  `http-types`, and `rand`'s custom-logger unsoundness. Re-check the list when
+  dependencies move.
 
 These are tracked in `AUDIT_EXECUTION_PLAN.md` (items C-2 through C-20).

deployment_assets/demo/evaluate_single_tenant/docker-compose.yml

@@ -44,7 +44,7 @@ services:
       - "8080:8080"
     environment:
       - RUST_LOG=info
-      - PANGOLIN_STORE_TYPE=memory
+      - PANGOLIN_STORAGE_TYPE=memory
       - PANGOLIN_NO_AUTH=true
       - PANGOLIN_ROOT_USER=root
       - PANGOLIN_ROOT_PASSWORD=rootpass

deployment_assets/helm/pangolin/Chart.yaml

@@ -2,8 +2,8 @@ apiVersion: v2
 name: pangolin
 description: A Helm chart for deploying the Pangolin Lakehouse Catalog
 type: application
-version: 0.6.0
-appVersion: "0.6.0"
+version: 0.7.0
+appVersion: "0.7.0"
 keywords:
   - iceberg
   - lakehouse

deployment_assets/production/azure_mongo/docker-compose.yml

@@ -16,7 +16,7 @@ services:
       - "8080:8080"
     environment:
       - RUST_LOG=info
-      - PANGOLIN_STORE_TYPE=mongo
+      - PANGOLIN_STORAGE_TYPE=mongo
       - DATABASE_URL=mongodb://mongo:27017
       - MONGO_DB_NAME=${MONGO_DB_NAME:-pangolin}
       - PANGOLIN_ROOT_USER=${PANGOLIN_ROOT_USER:-admin}

deployment_assets/production/azure_postgres/docker-compose.yml

@@ -25,7 +25,7 @@ services:
       - "8080:8080"
     environment:
       - RUST_LOG=info
-      - PANGOLIN_STORE_TYPE=postgres
+      - PANGOLIN_STORAGE_TYPE=postgres
       - DATABASE_URL=postgres://${POSTGRES_USER:-pangolin}:${POSTGRES_PASSWORD:-password}@postgres:5432/${POSTGRES_DB:-pangolin}
       - PANGOLIN_ROOT_USER=${PANGOLIN_ROOT_USER:-admin}
       - PANGOLIN_ROOT_PASSWORD=${PANGOLIN_ROOT_PASSWORD:-change-me-in-prod}

deployment_assets/production/gcp_mongo/docker-compose.yml

@@ -16,7 +16,7 @@ services:
       - "8080:8080"
     environment:
       - RUST_LOG=info
-      - PANGOLIN_STORE_TYPE=mongo
+      - PANGOLIN_STORAGE_TYPE=mongo
       - DATABASE_URL=mongodb://mongo:27017
       - MONGO_DB_NAME=${MONGO_DB_NAME:-pangolin}
       - PANGOLIN_ROOT_USER=${PANGOLIN_ROOT_USER:-admin}

deployment_assets/production/gcp_postgres/docker-compose.yml

@@ -25,7 +25,7 @@ services:
       - "8080:8080"
     environment:
       - RUST_LOG=info
-      - PANGOLIN_STORE_TYPE=postgres
+      - PANGOLIN_STORAGE_TYPE=postgres
       - DATABASE_URL=postgres://${POSTGRES_USER:-pangolin}:${POSTGRES_PASSWORD:-password}@postgres:5432/${POSTGRES_DB:-pangolin}
       - PANGOLIN_ROOT_USER=${PANGOLIN_ROOT_USER:-admin}
       - PANGOLIN_ROOT_PASSWORD=${PANGOLIN_ROOT_PASSWORD:-change-me-in-prod}

deployment_assets/production/s3_mongo/docker-compose.yml

@@ -16,7 +16,7 @@ services:
       - "8080:8080"
     environment:
       - RUST_LOG=info
-      - PANGOLIN_STORE_TYPE=mongo
+      - PANGOLIN_STORAGE_TYPE=mongo
       - DATABASE_URL=mongodb://mongo:27017
       - MONGO_DB_NAME=${MONGO_DB_NAME:-pangolin}
       - PANGOLIN_ROOT_USER=${PANGOLIN_ROOT_USER:-admin}

deployment_assets/production/s3_postgres/docker-compose.yml

@@ -25,7 +25,7 @@ services:
       - "8080:8080"
     environment:
       - RUST_LOG=info
-      - PANGOLIN_STORE_TYPE=postgres
+      - PANGOLIN_STORAGE_TYPE=postgres
       - DATABASE_URL=postgres://${POSTGRES_USER:-pangolin}:${POSTGRES_PASSWORD:-password}@postgres:5432/${POSTGRES_DB:-pangolin}
       - PANGOLIN_ROOT_USER=${PANGOLIN_ROOT_USER:-admin}
       - PANGOLIN_ROOT_PASSWORD=${PANGOLIN_ROOT_PASSWORD:-change-me-in-prod}

deployment_assets/test/localstack/docker-compose.yml

@@ -21,7 +21,7 @@ services:
       - "8080:8080"
     environment:
       - RUST_LOG=info
-      - PANGOLIN_STORE_TYPE=memory
+      - PANGOLIN_STORAGE_TYPE=memory
       - PANGOLIN_NO_AUTH=true
       - AWS_ACCESS_KEY_ID=test
       - AWS_SECRET_ACCESS_KEY=test

docker-compose.db-test.yml

@@ -42,3 +42,48 @@ services:
       interval: 5s
       timeout: 5s
       retries: 5
+
+  # Object storage for the file-IO half of the store compliance suite.
+  #
+  # This file used to provide only the databases, so `test_*_store_compliance`
+  # had no S3 to talk to: it fell through to the EC2 instance-metadata endpoint,
+  # hung for eleven seconds, and failed with a credentials error that looked
+  # nothing like "you have no object store". CI has always had one, so the
+  # documented local workflow and CI disagreed about what it took to run the
+  # suite.
+  #
+  # `warehouse` is the bucket the application itself expects; `bucket` and
+  # `test-bucket` are fixtures the compliance tests write to. Creating only
+  # `warehouse` (as `docker-compose.yml` does, for the app) leaves the suite
+  # failing with `NoSuchBucket`.
+  #
+  # The obvious image for this is `bitnami/minio`, whose MINIO_DEFAULT_BUCKETS
+  # makes the init container unnecessary. Bitnami withdrew their public Docker
+  # Hub images, so `bitnami/minio:latest` no longer resolves at all - it fails
+  # with `manifest unknown` rather than anything that names the cause.
+  minio:
+    image: minio/minio
+    environment:
+      MINIO_ROOT_USER: minioadmin
+      MINIO_ROOT_PASSWORD: minioadmin
+    command: server /data --console-address ":9001"
+    ports:
+      - "9000:9000"
+      - "9001:9001"
+    healthcheck:
+      test: ["CMD", "mc", "ready", "local"]
+      interval: 5s
+      timeout: 5s
+      retries: 20
+
+  createbuckets:
+    image: minio/mc
+    depends_on:
+      minio:
+        condition: service_healthy
+    entrypoint: >
+      /bin/sh -c "
+      /usr/bin/mc alias set m http://minio:9000 minioadmin minioadmin;
+      for b in warehouse bucket test-bucket; do /usr/bin/mc mb --ignore-existing m/$$b; done;
+      exit 0;
+      "

docker-compose.mongo-rs.yml

@@ -0,0 +1,41 @@
+# A single-node MongoDB replica set, for exercising the paths a standalone
+# `mongod` cannot reach.
+#
+# `docker-compose.db-test.yml` runs MongoDB standalone, which is the *degraded*
+# topology: no transactions and no retryable writes. Everything verified against
+# it therefore exercised the fallback paths only — and the fallback in
+# `delete_catalog` was itself broken for as long as the code had existed,
+# precisely because nothing ever ran the other branch.
+#
+# A single node is enough: a replica set of one supports transactions and
+# retryable writes, which is what distinguishes the two code paths. It is not a
+# model of production redundancy and is not meant to be.
+#
+#   docker compose -f docker-compose.mongo-rs.yml up -d
+#   export PANGOLIN_TEST_MONGO_URL='mongodb://localhost:27018/?replicaSet=rs0&directConnection=true'
+#   cargo test -p pangolin_store
+#
+# Note the port: 27018, so this can run alongside the standalone on 27017 and
+# the two topologies can be tested in one sitting.
+#
+# No authentication here. A replica set with auth needs a shared keyfile, which
+# is meaningful setup for a throwaway test fixture; the standalone compose file
+# covers the authenticated path.
+
+services:
+  mongo-rs:
+    image: mongo:7.0
+    command: ["mongod", "--replSet", "rs0", "--bind_ip_all", "--port", "27018"]
+    ports:
+      - "27018:27018"
+    healthcheck:
+      # Health is "the replica set has a primary", not "the process started".
+      # `mongod` accepts connections long before the set is initiated, and a
+      # client that connects in between fails in confusing ways.
+      test: >-
+        mongosh --port 27018 --quiet --eval
+        "try { rs.status().ok } catch (e) { rs.initiate({_id:'rs0',members:[{_id:0,host:'localhost:27018'}]}).ok }"
+      interval: 5s
+      timeout: 10s
+      retries: 20
+      start_period: 10s