fix(s3): user metadata (x-amz-meta-*) was discarded on write

CHANGELOG.md

@@ -18,6 +18,14 @@ All notable changes to **stunt** are documented here. The format is based on
 - **aws-s3-style: conditional requests.** `GET`/`HEAD` honor `If-Match`,
   `If-None-Match`, `If-Modified-Since`, and `If-Unmodified-Since`, returning
   `304` or `412`; `PUT`/`DELETE` honor the ETag conditions.
+- **aws-s3-style: user metadata.** `x-amz-meta-*` request headers are stored
+  with the object and echoed on GET/HEAD, with control-byte and 2 KB total
+  validation.
+
+### Engine
+
+- **Response headers keep the adapter's casing.** `x-amz-meta-*` is emitted
+  lowercase, as real S3 does, instead of Go's canonical form.
 
 ### Engine
 

CONFORMANCE.md

@@ -40,7 +40,7 @@ Behavior columns come in two kinds: **verified** (an official SDK was driven aga
 | [avalara-style](adapters/avalara-style/) | Avalara AvaTax REST API `2` | 8 | VM | — | — | [5](#avalara-style) | [3](#avalara-style) |
 | [aws-cognito-style](adapters/aws-cognito-style/) | Amazon Cognito Identity Provider API `2016-04-18` | 7 | VM | — | — | [6](#aws-cognito-style) | [3](#aws-cognito-style) |
 | [aws-iam-sts-style](adapters/aws-iam-sts-style/) | AWS STS + IAM API `2011-06-15` | 2 | SDK | aws-sdk-go-v2 @ v1.43.7 | 2 | [3](#aws-iam-sts-style) | [3](#aws-iam-sts-style) |
-| [aws-s3-style](adapters/aws-s3-style/) | Amazon S3 API `2006-03-01` | 8 | SDK | aws-sdk-go-v2 @ v1.43.7 | 6 | [5](#aws-s3-style) | [9](#aws-s3-style) |
+| [aws-s3-style](adapters/aws-s3-style/) | Amazon S3 API `2006-03-01` | 8 | SDK | aws-sdk-go-v2 @ v1.43.7 | 6 | [5](#aws-s3-style) | [10](#aws-s3-style) |
 | [azure-devops-style](adapters/azure-devops-style/) | Azure DevOps REST API `7.1` | 17 | VM | — | — | [8](#azure-devops-style) | [6](#azure-devops-style) |
 | [azure-servicebus-style](adapters/azure-servicebus-style/) | Azure Service Bus + Storage `2024-01-01` | 18 | VM | — | — | [6](#azure-servicebus-style) | [3](#azure-servicebus-style) |
 | [azure-storage-style](adapters/azure-storage-style/) | Azure Storage Blob REST API `2024-08-04` | 9 | VM | — | — | [6](#azure-storage-style) | [3](#azure-storage-style) |
@@ -2598,11 +2598,12 @@ behavior notes live in each adapter's README.
 - No browser form POST uploads (POST policy)
 - No ListMultipartUploads (GET /{bucket}?uploads)
 
-**Deviations** (9)
+**Deviations** (10)
 
 - ETags are MD5 hex (multipart MD5(binary-concat)-N)
 - Multipart 5 MiB minimum part size not enforced (small parts allowed)
 - DELETE of a missing bucket is an idempotent 204 (real S3: 404 NoSuchBucket)
+- x-amz-meta-* suffixes are lowercased and the first occurrence wins
 - Per-chunk STREAMING signatures not verified (header SigV4 only)
 - Streaming checksum trailers discarded (unsupported-checksum)
 - DELETE object against a missing bucket is 204 (real S3: 404 NoSuchBucket)

adapters/aws-s3-style/README.md

@@ -123,6 +123,23 @@ Two documented deviations: weak validators (`W/"..."`) compare as strong,
 and a `DELETE` or `GET`/`HEAD` against a missing bucket is `204`/`NoSuchKey`
 where real S3 returns `NoSuchBucket`.
 
+### User metadata
+
+Request headers with the `x-amz-meta-` prefix are stored with the object
+and echoed on `GET` and `HEAD` with status `200`. They are not returned on
+error responses or on `ListObjectsV2`, matching real S3.
+
+For multipart uploads, metadata is captured at `CreateMultipartUpload` and
+propagated at `CompleteMultipartUpload`; `UploadPart` metadata is ignored. A
+`PUT` without metadata clears whatever was stored before.
+
+Validation rejects `\r`, `\n`, NUL, and other C0 bytes except TAB with
+`400 InvalidArgument`, and caps total user metadata at 2048 bytes with
+`400 MetadataTooLarge`. Suffixes are lowercased and the first occurrence
+wins. Response headers are emitted with the adapter's own casing rather
+than Go's canonical form, because real S3 sends them lowercase and SDKs
+preserve the suffix case after stripping the prefix.
+
 ### Streaming uploads (aws-chunked)
 
 A default SDK `PutObject` signs its payload with
@@ -287,10 +304,11 @@ All errors use S3-shaped XML:
 | `InvalidAccessKeyId` | 403 | Access key is not the documented synthetic AKID |
 | `RequestTimeTooSkewed` | 403 | `x-amz-date` outside the ±15-minute window |
 | `XAmzContentSHA256Mismatch` | 400 | `x-amz-content-sha256` header does not match the body bytes |
-| `InvalidArgument` | 400 | `encoding-type` other than `url` on a list request; invalid `x-amz-content-sha256`; `partNumber` outside `1..10000` |
+| `InvalidArgument` | 400 | `encoding-type` other than `url` on a list request; invalid `x-amz-content-sha256`; `partNumber` outside `1..10000`; an `x-amz-meta-*` name is empty or its value holds a rejected control byte |
 | `InvalidPart` | 400 | CompleteMultipartUpload lists a part that was never uploaded, or whose ETag does not match |
 | `InvalidPartOrder` | 400 | CompleteMultipartUpload part list is not in ascending order |
 | `MalformedXML` | 400 | CompleteMultipartUpload body is not valid `CompleteMultipartUpload` XML |
+| `MetadataTooLarge` | 400 | `x-amz-meta-*` exceeds 2048 bytes total |
 | `IncompleteBody` | 400 | `aws-chunked` framing is malformed, truncated, or over the decoder limits |
 | `PreconditionFailed` | 412 | An `If-Match`/`If-None-Match`/`If-Unmodified-Since`/`If-Modified-Since` condition did not hold |
 | `MethodNotAllowed` | 405 | POST to an object without `?uploads`/`?uploadId` |

adapters/aws-s3-style/scripts/lib.star

@@ -759,16 +759,97 @@ def _upsert_object(bucket, key, raw, ct, etag, meta):
         "bid": bid,
         "contentType": ct,
         "etag": etag,
-        "metadata": meta,
         "lastModified": _unix_to_iso8601(now_unix),
         "lastModifiedUnix": now_unix,
         "size": len(raw),
+        "metadata": meta,
     }
     if obj_id != None and obj_id != "":
         oc.update(obj_id, doc)
     else:
         oc.insert(doc)
 
+# ====================================================================
+# User metadata (x-amz-meta-*)
+# ====================================================================
+# S3 user metadata: request headers with the x-amz-meta- prefix are stored
+# with the object and echoed back on GET/HEAD 200 only (never on 304/412,
+# List, or error responses). The first occurrence wins: the engine already
+# collapses duplicate wire headers to v[0] (headerMap), and the collect
+# loop below keeps the first suffix on post-lower collision as
+# defense-in-depth. Suffixes are lowercased ASCII-only (hand-rolled, not
+# str.lower(), so non-ASCII bytes pass through unfolded and are preserved
+# for size/echo). The byte sum len(suffix)+len(value) over all collected
+# entries must fit in 2048 (the AWS 2KB user-metadata total; the prefix is
+# excluded, measured post-dedup with byte len, so 2048 passes / 2049 fails).
+
+# _ascii_lower lowercases ASCII A-Z only, preserving every other byte
+# (unlike str.lower(), which would fold non-ASCII too).
+def _ascii_lower(s):
+    out = ""
+    for i in range(len(s)):
+        ch = s[i]
+        if ch >= "A" and ch <= "Z":
+            out = out + chr(ord(ch) + 32)
+        else:
+            out = out + ch
+    return out
+
+# _meta_bad_value returns True when s holds a byte real S3 rejects in user
+# metadata: CR, LF, NUL, any other C0 control except TAB, or DEL. (CR/LF
+# cannot arrive over HTTP — Go rejects them at the transport — so this is
+# defense-in-depth covered by inspection, not e2e.)
+def _meta_bad_value(s):
+    for i in range(len(s)):
+        o = ord(s[i])
+        if o == 9:
+            continue
+        if o < 32 or o == 127:
+            return True
+    return False
+
+# _collect_metadata gathers x-amz-meta-* request headers (iteration keys are
+# already the lowercase canonical form). Empty suffix or rejected bytes ->
+# 400 InvalidArgument; post-dedup byte total over 2048 -> 400
+# MetadataTooLarge. Empty values are allowed; spaces are preserved verbatim.
+# Returns (meta, None) on success or (None, error_response).
+def _collect_metadata(req):
+    headers = req.get("headers")
+    if headers == None:
+        return {}, None
+    meta = {}
+    total = 0
+    for k in headers.keys():
+        if not _has_prefix(k, "x-amz-meta-"):
+            continue
+        suffix = _ascii_lower(k[len("x-amz-meta-"):])
+        v = headers.get(k, "")
+        if v == None:
+            v = ""
+        v = str(v)
+        if suffix == "":
+            return None, _invalid_argument(k, v, "Metadata name must not be empty.")
+        if _meta_bad_value(suffix) or _meta_bad_value(v):
+            return None, _invalid_argument(k, v, "Metadata contains invalid characters.")
+        if suffix in meta:
+            continue
+        meta[suffix] = v
+        total = total + len(suffix) + len(v)
+    if total > 2048:
+        return None, _xml_error("MetadataTooLarge", "Your metadata headers exceed the maximum allowed metadata size.", "", 400)
+    return meta, None
+
+# _meta_response_headers merges stored user metadata into a GET/HEAD 200
+# response-header dict (suffixes were lowercased at collect time). Legacy
+# docs stored without a metadata field fall back to {}.
+def _meta_response_headers(obj, base):
+    meta = obj.get("metadata", {})
+    if meta == None:
+        meta = {}
+    for k in meta.keys():
+        base["x-amz-meta-" + k] = meta[k]
+    return base
+
 # ====================================================================
 # Multipart upload core
 # ====================================================================
@@ -855,11 +936,15 @@ def _mpu_create(req, bucket, key):
         ct = "application/octet-stream"
 
     upload_id = "mpu_" + str(store_kv_incr("s3", "mpu_seq"))
+    meta, merr = _collect_metadata(req)
+    if merr != None:
+        return merr
     store_collection("mpu_uploads").insert({
         "id": upload_id,
         "bucket": bucket,
         "key": key,
         "contentType": ct,
+        "metadata": meta,
         "initiatedUnix": clock.now_unix(),
     })
 
@@ -1087,7 +1172,13 @@ def _mpu_complete(req, bucket, key):
     ct = upload.get("contentType", "application/octet-stream")
     if ct == None or ct == "":
         ct = "application/octet-stream"
-    _upsert_object(bucket, key, full, ct, etag, {})
+    # The object's user metadata is the Create request's (stored on the
+    # upload row); Complete-request and UploadPart meta are ignored. Legacy
+    # uploads stored without a metadata field fall back to {}.
+    meta = upload.get("metadata", {})
+    if meta == None:
+        meta = {}
+    _upsert_object(bucket, key, full, ct, etag, meta)
 
     _mpu_discard(upload_id)
     store_collection("mpu_uploads").delete(upload_id)

adapters/aws-s3-style/scripts/objects.star

@@ -91,8 +91,12 @@ def on_put_object(req):
 
     # Content-derived ETag (MD5 of the verbatim bytes); the write path
     # (blob + metadata doc) is shared with CompleteMultipartUpload.
+    # User metadata replaces any previous value (PUT without meta clears).
     etag = _etag(raw)
-    _upsert_object(bucket, key, raw, ct, etag, {})
+    meta, merr = _collect_metadata(req)
+    if merr != None:
+        return merr
+    _upsert_object(bucket, key, raw, ct, etag, meta)
 
     return respond(200, "", {
         "ETag": '"' + etag + '"',
@@ -131,13 +135,13 @@ def on_get_object(req):
     if etag == None:
         etag = ""
 
-    return respond(200, content, {
+    return respond(200, content, _meta_response_headers(obj, {
         "Content-Type": ct,
         "ETag": '"' + etag + '"',
         "Last-Modified": _obj_last_modified_rfc1123(obj),
         "Content-Length": str(len(content)),
         "x-amz-request-id": _req_id(),
-    })
+    }))
 
 # on_head_object returns metadata headers only (no body).
 def on_head_object(req):
@@ -166,13 +170,13 @@ def on_head_object(req):
     if size == None:
         size = 0
 
-    return respond(200, "", {
+    return respond(200, "", _meta_response_headers(obj, {
         "Content-Type": ct,
         "ETag": '"' + etag + '"',
         "Last-Modified": _obj_last_modified_rfc1123(obj),
         "Content-Length": _to_int_str(size),
         "x-amz-request-id": _req_id(),
-    })
+    }))
 
 # on_delete_object removes an object, or aborts an in-progress multipart
 # upload when the request carries an uploadId. Returns 204.

conformance/matrix.json

@@ -9383,6 +9383,7 @@
         "ETags are MD5 hex (multipart MD5(binary-concat)-N)",
         "Multipart 5 MiB minimum part size not enforced (small parts allowed)",
         "DELETE of a missing bucket is an idempotent 204 (real S3: 404 NoSuchBucket)",
+        "x-amz-meta-* suffixes are lowercased and the first occurrence wins",
         "Per-chunk STREAMING signatures not verified (header SigV4 only)",
         "Streaming checksum trailers discarded (unsupported-checksum)",
         "DELETE object against a missing bucket is 204 (real S3: 404 NoSuchBucket)",

conformance/matrix.yaml

@@ -152,6 +152,7 @@ adapters:
       - "ETags are MD5 hex (multipart MD5(binary-concat)-N)"
       - "Multipart 5 MiB minimum part size not enforced (small parts allowed)"
       - "DELETE of a missing bucket is an idempotent 204 (real S3: 404 NoSuchBucket)"
+      - "x-amz-meta-* suffixes are lowercased and the first occurrence wins"
       - "Per-chunk STREAMING signatures not verified (header SigV4 only)"
       - "Streaming checksum trailers discarded (unsupported-checksum)"
       - "DELETE object against a missing bucket is 204 (real S3: 404 NoSuchBucket)"

internal/engine/adapter_dispatch.go

@@ -160,7 +160,7 @@ func (e *Engine) runHandler(
 
 	// Write headers.
 	for k, v := range resp.Headers {
-		w.Header().Set(k, v)
+		w.Header()[k] = []string{v}
 	}
 
 	status := resp.Status