fix(s3): ETag was SHA-256, which broke client-side hash validation in AWS SDKs - #106
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #103.
Real S3 uses the MD5 digest of the object bytes as the
ETagon non-multipart uploads. We returned SHA-256, so a 64-character hex digest went out where SDKs expect 32.GetObjectAsyncthen threw on every read withAmazonClientException: Expected hash not equal to calculated hash(viaHashStream.CalculateHash), even though the body itself was byte-correct. Any client that validates theETagagainst the content could not read an object back.Found by @jeremydixon22 while running the AWS .NET SDK (AWSSDK.S3 4.0.102.4) against the
aws-s3-styleadapter, and isolated to the ETag value by rewriting only that header in a reverse proxy: the same SDK read succeeded against all four pinned S3/Core version combinations.The stated reason for the SHA-256 stand-in was that the crypto module has no MD5. That became inaccurate in b6ca853 (2026-08-22), which added
crypto.md5for protocol checksums;adapters/sqs-style/scripts/lib.star:656-662has used it since. This adapter was never updated.What changed
UploadPartETags are nowcrypto.md5(raw).hex(MD5(binary-concat(part MD5s))) + "-" + N, matching real S3. Starlark'schr()emits UTF-8, so hex-decoding part digests inside Starlark would corrupt any byte>= 0x80; the binary concatenation runs in a new total builtin,crypto.md5_hex_concat(internal/starlark/cryptomod.go:44). It returnsNoneon any shape mismatch rather than raising, since a Starlark error surfaces as an unhandled 500 and handlers have notry.CompleteMultipartUploadcompares part ETags case-insensitively and strips quotes and aW/prefix on both sides, which also closes a bypass where an empty<ETag></ETag>skipped the comparison entirely.Breaking
ETags change from 64 hex characters to 32. Fixtures and assertions pinning the old digests need regenerating. Multipart uploads started before this lands and completed afterward will fail
400 InvalidPart, because their part digests were computed under the previous scheme; abort and restart those. Recorded inCHANGELOG.md.Verification
Through AWSSDK.S3 4.0.102.4 and a hand-signed raw HTTP request, MD5 of
hello:stunt adapter lint adapters/aws-s3-styleis clean andjust conformance-matrixregenerates with no drift.