Skip to content

[Bug] gemini-stage-0 creates the corpus GCS bucket but never grants the Discovery Engine service agent storage.objectViewer on it — gem4gov datastore import then fails every document with PERMISSION_DENIED #145

Description

@JohnHales

Bug Description

In gemini-stage-0, discovery-engine.tf creates a GCS bucket (google_storage_bucket.gemini_enterprise_gcs_bucket) for the document corpus when create_bucket = true, and cmek.tf grants the project's Discovery Engine service agent roles/cloudkms.cryptoKeyEncrypterDecrypter on the CMEK key. But nothing grants that service agent (service-<PROJECT_NUMBER>@gcp-sa-discoveryengine.iam.gserviceaccount.com) roles/storage.objectViewer (or objectUser) on the bucket it just created — iam.tf only binds project-level roles to the admin/user GROUPS, and there is no google_storage_bucket_iam_member for the DE service agent anywhere in the stage. As a result, when the operator later runs gem4gov datastore import --source-type=gcs --gcs-bucket=<that bucket>, the ImportDocuments long-running operation completes done: true / error: null but with failureCount == totalCount: every document fails with code 7 PERMISSION_DENIED — "Permission denied while checking the GCS content of gs://<bucket>/<file>". The data store indexes zero documents. Because gem4gov's import does not poll or surface the per-document result (the counts live only in the operation metadata / response.errorSamples), this looks like a silent success.

Environment and Deployment Context

  • Stellar Engine Version/Commit: main (gemini-stage-0/discovery-engine.tf + iam.tf + cmek.tf; gem4gov-cli/data_stores.py import_gcs_documents)
  • Deployment Type:
    • US Region Restricted (e.g., Access Policy constraint)
    • FedRAMP Medium
    • FedRAMP High
    • DoD IL4
    • DoD IL5
    • Stand-alone / Custom
  • Affected Component: blueprints/fedramp-high/gemini-enterprise/gemini-stage-0/iam.tf (no bucket-scoped grant for the Discovery Engine service agent) + discovery-engine.tf (google_storage_bucket.gemini_enterprise_gcs_bucket).

Steps to Reproduce

  1. Run gemini-stage-0 with a GCS data store and create_bucket = true so the stage creates …-<display-name>-data.
  2. Copy documents into that bucket.
  3. Run gem4gov datastore import --project-id <proj> --source-type=gcs --data-store-id <id> --gcs-bucket <that bucket> and press Enter for the root path.
  4. Poll the import operation: metadata.failureCount == metadata.totalCount, response.errorSamples[*].code == 7, message "Permission denied while checking the GCS content of gs://<bucket>/<file>". The data store's documents list is empty.

Expected Behavior

gemini-stage-0 should grant the Discovery Engine service agent read on the corpus bucket it creates, e.g. a google_storage_bucket_iam_member binding roles/storage.objectViewer (or roles/storage.objectUser) for serviceAccount:service-<num>@gcp-sa-discoveryengine.iam.gserviceaccount.com on google_storage_bucket.gemini_enterprise_gcs_bucket — mirroring the CMEK grant it already makes for the same agent. Then a corpus import from that bucket succeeds without a manual IAM step.

Actual Behavior

The stage creates the bucket and the CMEK grant but not the storage grant, so every subsequent document import fails PERMISSION_DENIED and the data store indexes nothing — with no error surfaced by the CLI.

Additional Context

The KMS grant for the same agent is already present (google_kms_crypto_key_iam_member.discoveryengine_sa_kms_access), so the missing piece is a single parallel storage binding. Related: gem4gov's import is fire-and-forget with no polling/validation, which is what makes this failure silent — worth its own hardening (surface failureCount/errorSamples, or poll the LRO).


Activity

  1. added
    BugSomething isn't working
    Priority - MediumStandard features and non-blocking bugs; important for the current milestone but not urgent
    Level of Effort - LowQuick, well-defined tasks with no unknowns; takes a few hours up to one day to complete
    on Jul 10, 2026
  2. added theissue type on Jul 10, 2026
  3. JohnHales commented on Aug 17, 2026

    @JohnHales
    CollaboratorAuthor

    Still reproduces at v3.0.0 (f64ce6cd). Re-verified by inspection of gemini-stage-0:

    • There is no bucket-scoped IAM binding anywhere in the stage — no google_storage_bucket_iam_member, and no storage.objectViewer / objectUser / admin grant in any .tf file.
    • The Discovery Engine service agent is bound in exactly one place in the whole stage: cmek.tf:29, the roles/cloudkms.cryptoKeyEncrypterDecrypter grant on the CMEK key. The parallel storage grant is still absent.

    One thing that has changed since this was filed, and it makes the fix slightly larger than "add one binding": the corpus bucket is now created with for_each over gcs_data_store_configs (discovery-engine.tf:58), so a deployment with several GCS data stores creates several buckets. Each one needs the grant, so the fix wants to be a google_storage_bucket_iam_member with the same for_each rather than a single binding.

    Worth noting alongside: because gem4gov datastore import is fire-and-forget, this failure still presents as a silent success — the operation reports done: true with error: null while failureCount == totalCount. Anyone verifying a fix should read metadata.successCount / failureCount rather than the operation's error field, which is misleading here.

  4. added a commit that references this issue on Sep 6, 2026
    69b9a58
  5. aghassemlouei commented on Oct 5, 2026

    @aghassemlouei
    Member

    Thanks for tracking down the missing Discovery Engine service agent IAM binding on the corpus GCS bucket, @JohnHales! We added the roles/storage.objectViewer grant for the Discovery Engine service agent in #219 (commit 69b9a582) and refined the multi-bucket loop in #274 (commit c7353a39). Please feel free to validate document imports on main whenever convenient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugSomething isn't workingGemini - GovernmentGemini for Government (G4G) relatedLevel of Effort - LowQuick, well-defined tasks with no unknowns; takes a few hours up to one day to completePriority - MediumStandard features and non-blocking bugs; important for the current milestone but not urgent

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions