fix(s3): user metadata (x-amz-meta-*) was discarded on write - #109
Merged
Merged
Conversation
# Conflicts: # adapters/aws-s3-style/scripts/lib.star # adapters/aws-s3-style/scripts/objects.star # internal/engine/aws_s3_style_test.go
# Conflicts: # CHANGELOG.md # adapters/aws-s3-style/scripts/lib.star # adapters/aws-s3-style/scripts/objects.star
# Conflicts: # CHANGELOG.md # CONFORMANCE.md # adapters/aws-s3-style/README.md # conformance/matrix.json # conformance/matrix.yaml # internal/engine/aws_s3_style_test.go
# Conflicts: # CONFORMANCE.md # adapters/aws-s3-style/README.md # conformance/matrix.json # conformance/matrix.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #105.
A
PUTcarryingx-amz-meta-kind: samplesucceeded and every subsequent read lost the value.GetObjectMetadatareported nokindand a rawGETreturned nox-amz-meta-kindheader, whileContent-Typesurvived. Found by @jeremydixon22 driving theaws-s3-styleadapter with AWSSDK.S3 4.0.102.4.Two independent defects, and fixing only the first is not enough for SDK clients.
Root cause
Storage. The write path persisted a fixed document shape;
adapters/aws-s3-style/scripts/lib.starstoredbucket,key,bid,contentType,etag, and timestamps, andon_put_objectread onlyContent-Typeoff the request. The read paths returned a fixed header set, so nothing could echo user metadata even if it had been stored.Header casing. The engine wrote response headers with
w.Header().Set, which canonicalizesx-amz-meta-kindtoX-Amz-Meta-Kind. Real S3 emits these lowercase. The .NET SDK strips thex-amz-meta-prefix and preserves the remaining case, so a client reading backkindreceivedKindand its ordinal lookup failed withKeyNotFoundException. The AWS SDK matrix reproduced this across all four pinned S3/Core version combinations.The casing half was not caught by adapter-level Go tests, since they compare against header maps that canonicalize the lookup. It only surfaced when a real SDK read the wire bytes.
What changed
x-amz-meta-*request headers are collected, stored on the object document, and echoed onGETandHEADwith status200. They are not returned on error responses or onListObjectsV2, matching S3.CreateMultipartUploadpropagates atCompleteMultipartUpload.UploadPartandComplete-request metadata are ignored. APUTwithout metadata clears what was stored before.\r,\n, NUL, and other C0 bytes except TAB with400 InvalidArgument, and caps total user metadata at 2048 bytes with400 MetadataTooLarge. Suffixes are lowercased and the first occurrence wins.internal/engine/adapter_dispatch.go:162andinternal/engine/engine.go:622now assign into the header map directly rather than callingSet, so adapter-supplied header names reach the wire unchanged. FixedContent-Typedefaults still useSet.Verification
TestHeaderCase*asserts on the raw header map, notHeader().Get, becauseGetcanonicalizes the lookup key and would mask the defect.Through AWSSDK.S3 4.0.102.4:
stunt adapter lint adapters/aws-s3-styleis clean andjust conformance-matrixregenerates with no drift.