Migrate Azure blob storage to azure_storage_blob 1.0.0 - #6693
Migrate Azure blob storage to azure_storage_blob 1.0.0#6693siva-abstract-security wants to merge 6 commits into
Conversation
Move the workspace off the legacy Azure SDK and onto the 1.0 line: azure_core 1.1, azure_identity 1.0 and azure_storage_blob 1.0. `azure_storage` is dropped outright. The rewritten SDK has no successor for `StorageCredentials`, `CloudLocation` or `ConnectionString`, so the concepts it provided have to be rebuilt on top of the pipeline instead of renamed. Feature names changed with the rewrite: `enable_reqwest_rustls` is now `reqwest_rustls`, and the `azurite_workaround` features no longer exist, so they leave `integration-testsuite`. `hmac_rust` survives in `azure_core` 1.1, which matters because a shared key signing policy needs it. The new SDK resolves to a smaller graph: `Cargo.lock` loses 296 lines net. This commit only moves the dependencies. `quickwit-storage` does not build against them yet.
…election Two capabilities the rewritten Azure SDK no longer provides, added ahead of porting the blob storage backend itself. `azure_shared_key` signs requests with the storage account key. The 1.0 SDK authenticates with Entra ID tokens only, and the SDK team has said shared key support will not return (Azure/azure-sdk-for-rust#2975). Quickwit documents `azure.access_key` as a supported credential, and Azurite accepts shared key only, so the signing `azure_storage` 0.21 used to provide lives here now. The policy runs per retry, because the service rejects an `x-ms-date` that has drifted more than fifteen minutes and a retried request would otherwise carry a stale timestamp. `azure_credentials` replaces `azure_identity::create_credential()`, which no longer exists: the 1.0 line removed `DefaultAzureCredential` along with it, and the remaining `DeveloperToolsCredential` chains the two CLIs only. Workload identity is chosen when all three variables the webhook injects are present, managed identity otherwise, and `AZURE_CREDENTIAL_KIND` still pins the choice explicitly. The container client is built here too, since 1.0 clients take a container URL rather than an account name plus a cloud location, which removes the special case a sovereign endpoint used to need. Both modules compile and carry unit tests. The backend in `azure_blob_storage.rs` is not ported yet, so the crate still does not build.
Rewrites `AzureBlobStorage` against the 1.0 clients and deletes the last references to the legacy SDK, so `quickwit-storage` builds again. The client mapping is mostly mechanical: `ContainerClient` becomes `BlobContainerClient`, `put_block_blob` becomes `BlockBlobClient::upload`, `put_block` and `put_block_list` become `stage_block` and `commit_block_list`, and `list_blobs` yields a `Pager` rather than a `Pageable`. Two places needed more thought. Downloads no longer walk a page of chunk responses. `BlobClient::download` returns one result whose `body` is a stream, so `copy_to` and `get_slice_stream` share a single `get_to_reader` helper that pulls the first chunk before returning. That keeps an error arriving with the response headers inside the retry rather than handing it to a caller with no way to retry. `BlockBlobClient` is not `Clone` in 1.0, so each part of a multipart upload builds its own client from the container client. Construction is local: the pipeline is behind an `Arc` and only the URL differs. Two behaviour notes. Single part upload sets `blob_content_md5` rather than a transactional checksum, because the partitioned upload path does not expose one, so the digest is stored with the blob instead of verified per request. Multipart still checks per block via `stage_block`. `ClientBuilder::emulator()` is gone, and the SDK could not have kept it, since the emulator authenticates with a shared key. The integration test now asks this crate to create and delete its container so the signing policy stays internal. 85 unit tests pass, clippy is clean, and `quickwit-cli` builds under `release-feature-set`.
Verified against Azurite, which rejected `commit_block_list` with `AuthorizationFailure` while every other operation authorized fine. The generated operations are inconsistent about `Content-Length`. `stage_block` inserts the header itself, so signing saw it. `commit_block_list` leaves it to the transport, so signing saw nothing and covered an empty length while the wire carried the real one, and shared key rejects that mismatch. The response says only that the signature is malformed, so the operation-specific nature of the failure is invisible from the error. Set the header before signing when the body length is known and the header is absent, which makes the signature and the wire agree whatever the operation did. A zero length body still signs an empty slot, as the specification requires from API version 2015-02-21 onwards. Two tests cover it through `Policy::send` with a capturing terminal policy, one for the header being added and one for an existing value being left alone. Removing the fix fails the first and nothing else. Azurite needed two changes to run the suite at all. The pinned 3.24.0 predates the API version `azure_storage_blob` 1.0 sends, and so does 3.36.0, the newest released, so the emulator now runs with `--skipApiVersionCheck`. Skipping the check keeps the emulated client sending the same request as production, which is what the signing needs to be tested against, rather than pinning an older API version only the tests would use.
siva-abstract-security
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3f8944b283
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| _ => { | ||
| if workload_identity_env_is_complete() { | ||
| info!("using azure workload identity credential"); | ||
| return Ok(WorkloadIdentityCredential::new(None)?); | ||
| } | ||
| info!("using azure managed identity credential"); | ||
| Ok(ManagedIdentityCredential::new(None)?) |
There was a problem hiding this comment.
When Azure authentication is configured with AZURE_CLIENT_ID, AZURE_TENANT_ID, and AZURE_CLIENT_SECRET, the workload-identity check fails because there is no federated token file, and this fallback always constructs a managed-identity credential, which ignores the client secret and attempts IMDS instead. The previous azure_identity::create_credential() path explicitly supported environment credentials, so deployments using service principals will lose Azure Blob access after this upgrade; select a client-secret/environment credential before falling back to managed identity.
Useful? React with 👍 / 👎.
| info!("using azure managed identity credential"); | ||
| Ok(ManagedIdentityCredential::new(None)?) |
There was a problem hiding this comment.
When a deployment uses a user-assigned managed identity by setting AZURE_CLIENT_ID without workload-identity variables, this branch selects managed identity but passes no credential options, discarding the requested client ID. It consequently requests the system-assigned identity instead, so hosts with only the user-assigned identity cannot authenticate and hosts with both identities may use the wrong principal; construct the managed-identity options from AZURE_CLIENT_ID before creating the credential.
Useful? React with 👍 / 👎.
`BlockBlobClient::upload()` only exposes `blob_content_md5`, which the service stores as a property without checking it against the body, so the port had quietly dropped the integrity check `put_block_blob(..).hash(..)` used to give us. Splits are immutable and never re-verified, so a corrupted upload would have been permanent and silent. Stage a single block and commit it instead. `stage_block` takes a transactional checksum, so the service rejects a payload that does not match on arrival. The cost is one extra request per object below the multipart threshold, which is the cheaper side of this trade. Also pins the path encoding in the canonicalized resource, which a review question prompted me to check. The path is signed exactly as the URI carries it, escapes and all, while query parameters are decoded. That asymmetry is specified: "any portion of the CanonicalizedResource string that is derived from the resource's URI should be encoded exactly as it is in the URI", and the query steps separately say to URL-decode each name and value. Decoding the path instead is rejected with `AuthorizationFailure`, confirmed against Azurite. Two tests now hold that shape in place so it does not get tidied away later.
Two credential regressions from replacing `azure_identity::create_credential()`, both raised in review. The old chain tried an environment credential before managed identity, reading `AZURE_TENANT_ID`, `AZURE_CLIENT_ID` and `AZURE_CLIENT_SECRET`. The replacement recognized only workload identity and fell through to managed identity, so a deployment authenticating with a service principal had its secret ignored and its request sent to IMDS. Those deployments would have lost Azure access on upgrade, with an error naming neither the secret nor the reason. Separately, `AZURE_CLIENT_ID` on its own names a user-assigned managed identity. Passing no options asks IMDS for the system-assigned identity, which is either absent or, on a host carrying both, the wrong principal. The client id now becomes `UserAssignedId::ClientId`. Selection moved into `select_token_credential_kind`, which takes a lookup function rather than reading the process environment, so precedence is covered by ordinary tests. Mutating environment variables inside a test is not safe while other tests run, and precedence between overlapping variable sets is exactly what needs pinning: a secret beats a federated token file, a partial set of either falls back rather than half-configuring a credential, and a blank value counts as unset. `TokenCredentialKind` carries the secret, so its `Debug` is hand written to redact it. A derived one would print the secret, and this type appears in test failure output.
There was a problem hiding this comment.
💡 Codex Review
https://github.com/quickwit-oss/quickwit/blob/abb142913b283f85fdb5b46d04eb77bf8281a6a0/quickwit-storage/src/object_storage/azure_blob_storage.rs#L349
When two small put calls target the same blob concurrently, both stage their payload under the fixed block:00000 ID. Because uncommitted blocks are keyed by blob and block ID, the later stage replaces the earlier payload; the earlier commit can then report success after publishing the other writer's bytes, while the remaining commit may fail because its uncommitted block was consumed. The previous single-request upload path did not introduce this cross-write race, so generate a unique block ID once per upload, outside the retry closure.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
1 participant
migrates azure blob storage to the rewritten sdk - azure_core 1.1, azure_identity 1.0, azure_storage_blob 1.0.
azure_storageis gone entirely and nothing replaces it. closes #6672.the upgrade fixes the bug on its own -
azure_identity1.0 re-reads the federated token file once the cached copy is >600s old, so the assertion at T+24h is fresh and the indexer stops dying. no workaround needed.two things 1.0 dropped that i had to rebuild:
1 - shared key signing. 1.0 does entra tokens only and MS says it's not coming back (Azure/azure-sdk-for-rust#2975). we document
access_keyand azurite speaks shared key only, soazure_shared_keydoes whatazure_storage0.21 used to.2 - credential selection.
create_credential()andDefaultAzureCredentialare both gone, soazure_credentialspicks workload vs managed identity from the env explicitly.one thing worth your attention - the generated ops disagree about
Content-Length.stage_blocksets the header,commit_block_listleaves it to the transport, so signing covered an empty length while the wire carried a real one and shared key rejected it asAuthorizationFailurewith nothing pointing at why. only azurite caught this - unit tests over the string construction didn't.azurite also needs
--skipApiVersionChecknow. 3.24.0 and even 3.36.0 (newest released) both predate the api version 1.0 sends. docker-compose carries that plus the image bump.green: 80 unit tests, the full azurite integration suite, clippy, and
quickwit-cliunderrelease-feature-set.what i don't know:
1 - haven't run this against a real azure account, only azurite. so managed identity and workload identity never actually executed, including the 24h refresh this closes.
2 - single part upload sets
blob_content_md5instead of a transactional checksum, since the partitioned upload path doesn't expose one - stored with the blob rather than checked per request. multipart still checks per block viastage_block.3 - whether
--skipApiVersionCheckis hiding a real incompatibility. azurite took every op the suite runs, but that flag stops it telling us what it doesn't implement.written with claude opus 5.