Skip to content

fix(s3): user metadata (x-amz-meta-*) was discarded on write - #109

Merged
deblasis merged 6 commits into
mainfrom
pr/105-metadata
Oct 1, 2026
Merged

deblasis merged 6 commits into
mainfrom
pr/105-metadata

Conversation

@deblasis

@deblasis deblasis commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Fixes #105.

A PUT carrying x-amz-meta-kind: sample succeeded and every subsequent read lost the value. GetObjectMetadata reported no kind and a raw GET returned no x-amz-meta-kind header, while Content-Type survived. Found by @jeremydixon22 driving the aws-s3-style adapter with AWSSDK.S3 4.0.102.4.

Two independent defects, and fixing only the first is not enough for SDK clients.

Root cause

Storage. The write path persisted a fixed document shape; adapters/aws-s3-style/scripts/lib.star stored bucket, key, bid, contentType, etag, and timestamps, and on_put_object read only Content-Type off the request. The read paths returned a fixed header set, so nothing could echo user metadata even if it had been stored.

Header casing. The engine wrote response headers with w.Header().Set, which canonicalizes x-amz-meta-kind to X-Amz-Meta-Kind. Real S3 emits these lowercase. The .NET SDK strips the x-amz-meta- prefix and preserves the remaining case, so a client reading back kind received Kind and its ordinal lookup failed with KeyNotFoundException. The AWS SDK matrix reproduced this across all four pinned S3/Core version combinations.

The casing half was not caught by adapter-level Go tests, since they compare against header maps that canonicalize the lookup. It only surfaced when a real SDK read the wire bytes.

What changed

  • x-amz-meta-* request headers are collected, stored on the object document, and echoed on GET and HEAD with status 200. They are not returned on error responses or on ListObjectsV2, matching S3.
  • Multipart: metadata captured at CreateMultipartUpload propagates at CompleteMultipartUpload. UploadPart and Complete-request metadata are ignored. A PUT without metadata clears what was stored before.
  • Validation rejects \r, \n, NUL, and other C0 bytes except TAB with 400 InvalidArgument, and caps total user metadata at 2048 bytes with 400 MetadataTooLarge. Suffixes are lowercased and the first occurrence wins.
  • internal/engine/adapter_dispatch.go:162 and internal/engine/engine.go:622 now assign into the header map directly rather than calling Set, so adapter-supplied header names reach the wire unchanged. Fixed Content-Type defaults still use Set.

Verification

$ env -u GOROOT go test -race ./internal/engine -run 'TestAwsS3|TestAWSS3|TestHeaderCase' -v
--- PASS: TestAwsS3StyleAdapter
--- PASS: TestAwsS3StyleSigV4Verification
--- PASS: TestAwsS3StyleMultipartUpload
--- PASS: TestAWSS3StyleBinaryRoundTrip
--- PASS: TestAWSS3Metadata
--- PASS: TestHeaderCaseApplyDecisionPreservesLowercase
--- PASS: TestHeaderCaseRunHandlerPreservesLowercase

TestHeaderCase* asserts on the raw header map, not Header().Get, because Get canonicalizes the lookup key and would mask the defect.

Through AWSSDK.S3 4.0.102.4:

{ "customMetadataPreserved": true, "contentHashMetadataPreserved": true, "getCustomMetadataPreserved": true }

stunt adapter lint adapters/aws-s3-style is clean and just conformance-matrix regenerates with no drift.

# Conflicts:
#	adapters/aws-s3-style/scripts/lib.star
#	adapters/aws-s3-style/scripts/objects.star
#	internal/engine/aws_s3_style_test.go
# Conflicts:
#	CHANGELOG.md
#	adapters/aws-s3-style/scripts/lib.star
#	adapters/aws-s3-style/scripts/objects.star
# Conflicts:
#	CHANGELOG.md
#	CONFORMANCE.md
#	adapters/aws-s3-style/README.md
#	conformance/matrix.json
#	conformance/matrix.yaml
#	internal/engine/aws_s3_style_test.go
# Conflicts:
#	CONFORMANCE.md
#	adapters/aws-s3-style/README.md
#	conformance/matrix.json
#	conformance/matrix.yaml
@deblasis
deblasis merged commit 57796b6 into main Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S3 user metadata (x-amz-meta-*) was discarded on write

1 participant