fix(engine): aws-chunked uploads stored the SigV4 framing as object content

CHANGELOG.md

@@ -19,6 +19,11 @@ All notable changes to **stunt** are documented here. The format is based on
   `If-None-Match`, `If-Modified-Since`, and `If-Unmodified-Since`, returning
   `304` or `412`; `PUT`/`DELETE` honor the ETag conditions.
 
+### Engine
+
+- **aws-chunked request bodies are decoded before dispatch.** SigV4 streaming
+  uploads from AWS SDKs store the object bytes instead of the chunk framing.
+
 ## [0.52.0] — 2026-08-24
 
 The conformance campaign: every real API adapter now carries a real test

CONFORMANCE.md

@@ -40,7 +40,7 @@ Behavior columns come in two kinds: **verified** (an official SDK was driven aga
 | [avalara-style](adapters/avalara-style/) | Avalara AvaTax REST API `2` | 8 | VM | — | — | [5](#avalara-style) | [3](#avalara-style) |
 | [aws-cognito-style](adapters/aws-cognito-style/) | Amazon Cognito Identity Provider API `2016-04-18` | 7 | VM | — | — | [6](#aws-cognito-style) | [3](#aws-cognito-style) |
 | [aws-iam-sts-style](adapters/aws-iam-sts-style/) | AWS STS + IAM API `2011-06-15` | 2 | SDK | aws-sdk-go-v2 @ v1.43.7 | 2 | [3](#aws-iam-sts-style) | [3](#aws-iam-sts-style) |
-| [aws-s3-style](adapters/aws-s3-style/) | Amazon S3 API `2006-03-01` | 8 | SDK | aws-sdk-go-v2 @ v1.43.7 | 6 | [5](#aws-s3-style) | [7](#aws-s3-style) |
+| [aws-s3-style](adapters/aws-s3-style/) | Amazon S3 API `2006-03-01` | 8 | SDK | aws-sdk-go-v2 @ v1.43.7 | 6 | [5](#aws-s3-style) | [9](#aws-s3-style) |
 | [azure-devops-style](adapters/azure-devops-style/) | Azure DevOps REST API `7.1` | 17 | VM | — | — | [8](#azure-devops-style) | [6](#azure-devops-style) |
 | [azure-servicebus-style](adapters/azure-servicebus-style/) | Azure Service Bus + Storage `2024-01-01` | 18 | VM | — | — | [6](#azure-servicebus-style) | [3](#azure-servicebus-style) |
 | [azure-storage-style](adapters/azure-storage-style/) | Azure Storage Blob REST API `2024-08-04` | 9 | VM | — | — | [6](#azure-storage-style) | [3](#azure-storage-style) |
@@ -2598,11 +2598,13 @@ behavior notes live in each adapter's README.
 - No browser form POST uploads (POST policy)
 - No ListMultipartUploads (GET /{bucket}?uploads)
 
-**Deviations** (7)
+**Deviations** (9)
 
 - ETags are MD5 hex (multipart MD5(binary-concat)-N)
 - Multipart 5 MiB minimum part size not enforced (small parts allowed)
 - DELETE of a missing bucket is an idempotent 204 (real S3: 404 NoSuchBucket)
+- Per-chunk STREAMING signatures not verified (header SigV4 only)
+- Streaming checksum trailers discarded (unsupported-checksum)
 - DELETE object against a missing bucket is 204 (real S3: 404 NoSuchBucket)
 - GET/HEAD object against a missing bucket is 404 NoSuchKey (real S3: 404 NoSuchBucket)
 - SigV4 canonical URI/query rebuilt from decoded values — duplicates indistinguishable

adapters/aws-s3-style/README.md

@@ -123,6 +123,24 @@ Two documented deviations: weak validators (`W/"..."`) compare as strong,
 and a `DELETE` or `GET`/`HEAD` against a missing bucket is `204`/`NoSuchKey`
 where real S3 returns `NoSuchBucket`.
 
+### Streaming uploads (aws-chunked)
+
+A default SDK `PutObject` signs its payload with
+`STREAMING-AWS4-HMAC-SHA256-PAYLOAD` and wraps the bytes in
+`Content-Encoding: aws-chunked` framing. The engine decodes that framing
+before rules, profiles, and handlers run, so the adapter stores the object
+bytes. `Content-Encoding` and `x-amz-content-sha256` pass through untouched,
+since SigV4 signs them.
+
+Framing that is malformed, truncated, or over the limits below returns
+`400 IncompleteBody`; decoded output over `max_body_bytes` returns `413`.
+Limits are 4 KiB per chunk or trailer line, 10,000 data chunks, and 32
+trailers totalling 8 KiB. `x-amz-decoded-content-length` is verified when
+exactly one valid value is present and ignored otherwise.
+
+Two documented deviations: per-chunk signatures are not verified (the
+header signature is), and streaming checksum trailers are discarded.
+
 ## Auth — AWS Signature Version 4 (SigV4), verified for real
 
 Amazon S3 uses **AWS Signature Version 4** (SigV4) for authentication. This
@@ -273,6 +291,7 @@ All errors use S3-shaped XML:
 | `InvalidPart` | 400 | CompleteMultipartUpload lists a part that was never uploaded, or whose ETag does not match |
 | `InvalidPartOrder` | 400 | CompleteMultipartUpload part list is not in ascending order |
 | `MalformedXML` | 400 | CompleteMultipartUpload body is not valid `CompleteMultipartUpload` XML |
+| `IncompleteBody` | 400 | `aws-chunked` framing is malformed, truncated, or over the decoder limits |
 | `PreconditionFailed` | 412 | An `If-Match`/`If-None-Match`/`If-Unmodified-Since`/`If-Modified-Since` condition did not hold |
 | `MethodNotAllowed` | 405 | POST to an object without `?uploads`/`?uploadId` |
 | `NoSuchBucket` | 404 | Bucket doesn't exist |

conformance/matrix.json

@@ -9383,6 +9383,8 @@
         "ETags are MD5 hex (multipart MD5(binary-concat)-N)",
         "Multipart 5 MiB minimum part size not enforced (small parts allowed)",
         "DELETE of a missing bucket is an idempotent 204 (real S3: 404 NoSuchBucket)",
+        "Per-chunk STREAMING signatures not verified (header SigV4 only)",
+        "Streaming checksum trailers discarded (unsupported-checksum)",
         "DELETE object against a missing bucket is 204 (real S3: 404 NoSuchBucket)",
         "GET/HEAD object against a missing bucket is 404 NoSuchKey (real S3: 404 NoSuchBucket)",
         "SigV4 canonical URI/query rebuilt from decoded values — duplicates indistinguishable",

conformance/matrix.yaml

@@ -152,6 +152,8 @@ adapters:
       - "ETags are MD5 hex (multipart MD5(binary-concat)-N)"
       - "Multipart 5 MiB minimum part size not enforced (small parts allowed)"
       - "DELETE of a missing bucket is an idempotent 204 (real S3: 404 NoSuchBucket)"
+      - "Per-chunk STREAMING signatures not verified (header SigV4 only)"
+      - "Streaming checksum trailers discarded (unsupported-checksum)"
       - "DELETE object against a missing bucket is 204 (real S3: 404 NoSuchBucket)"
       - "GET/HEAD object against a missing bucket is 404 NoSuchKey (real S3: 404 NoSuchBucket)"
       - "SigV4 canonical URI/query rebuilt from decoded values — duplicates indistinguishable"

internal/engine/aws_s3_style_test.go

@@ -96,9 +96,15 @@ func awsSigV4SigningKey(secret, date, region, service string) []byte {
 // x-amz-date are signed and the payload hash covers the body bytes.
 func awsSigV4Sign(t *testing.T, req *http.Request, body []byte, service, accessKey, secretKey string, at time.Time) {
 	t.Helper()
+	awsSigV4SignPayload(t, req, awsSHA256Hex(body), service, accessKey, secretKey, at)
+}
+
+// awsSigV4SignPayload signs req in place with real SigV4 using an explicit
+// payload hash. Streaming (aws-chunked) uploads sign the STREAMING literal
+// rather than the body bytes, so chunked tests pass the literal here.
+func awsSigV4SignPayload(t *testing.T, req *http.Request, payloadHash, service, accessKey, secretKey string, at time.Time) {
 	amzDate := at.UTC().Format("20060102T150405Z")
 	date := amzDate[:8]
-	payloadHash := awsSHA256Hex(body)
 	req.Header.Set("x-amz-date", amzDate)
 
 	signedHeaders := []string{"host", "x-amz-date"}

internal/engine/chunked_aws.go

@@ -0,0 +1,255 @@
+// AWS-chunked (SigV4 STREAMING) framing decode.
+//
+// S3 SDKs using SigV4 streaming uploads send the object bytes wrapped in
+// aws-chunked framing: hex-size chunk lines with opaque ";extensions",
+// strict CRLF delimiters, a terminal zero chunk, and optional trailers.
+// The engine strips that framing before adapter dispatch so handlers see
+// the decoded bytes, while the Content-Encoding and x-amz-content-sha256
+// headers are preserved untouched (SigV4 signs them).
+package engine
+
+import (
+	"bytes"
+	"net/http"
+	"strconv"
+	"strings"
+)
+
+const (
+	// awsChunkedReadOverhead is the headroom above bodyLimit granted to
+	// the encoded (still framed) read: framing, extensions, and trailers
+	// for up to awsChunkedMaxChunks chunks.
+	awsChunkedReadOverhead = int64(2 << 20) // 2 MiB
+	// awsChunkedMaxLine caps one chunk-size or trailer line (incl CRLF).
+	awsChunkedMaxLine = 4096
+	// awsChunkedMaxChunks caps data chunks per request.
+	awsChunkedMaxChunks = 10000
+	// awsChunkedMaxTrailers caps trailer header lines after the zero chunk.
+	awsChunkedMaxTrailers = 32
+	// awsChunkedMaxTrailerBytes caps total trailer line bytes (incl CRLF).
+	awsChunkedMaxTrailerBytes = 8 << 10
+)
+
+// awsChunkedError is a decode failure: status is 400 (framing) or 413
+// (decoded/encoded over limit). The 400 body carries the S3 IncompleteBody
+// code; the error is never echoed back with request bytes.
+type awsChunkedError struct {
+	status int
+}
+
+func (e *awsChunkedError) Error() string {
+	if e.status == http.StatusRequestEntityTooLarge {
+		return "aws-chunked decoded body exceeds limit"
+	}
+	return "IncompleteBody"
+}
+
+// writeIncompleteBody writes the S3 IncompleteBody error envelope for
+// aws-chunked framing failures. The request bytes are never echoed.
+func writeIncompleteBody(w http.ResponseWriter) {
+	w.Header().Set("Content-Type", "application/xml")
+	w.WriteHeader(http.StatusBadRequest)
+	_, _ = w.Write([]byte(`<?xml version="1.0" encoding="UTF-8"?><Error><Code>IncompleteBody</Code><Message>The request body terminated unexpectedly or did not match the expected length.</Message></Error>`))
+}
+
+// flattenHeaderTokens flattens multi-value headers into comma-split,
+// OWS-trimmed tokens (empty members dropped per RFC 9110 list parsing).
+func flattenHeaderTokens(vals []string) []string {
+	var out []string
+	for _, v := range vals {
+		for _, tok := range strings.Split(v, ",") {
+			tok = strings.Trim(tok, " \t")
+			if tok == "" {
+				continue
+			}
+			out = append(out, tok)
+		}
+	}
+	return out
+}
+
+// hasAwsChunkedToken reports whether any Content-Encoding value carries
+// the aws-chunked token (comma-split, OWS-trimmed, case-folded).
+func hasAwsChunkedToken(vals []string) bool {
+	for _, tok := range flattenHeaderTokens(vals) {
+		if strings.ToLower(tok) == "aws-chunked" {
+			return true
+		}
+	}
+	return false
+}
+
+// hasStreamingPayloadToken reports whether any x-amz-content-sha256 value
+// is a STREAMING-* token (malformed-client trigger without Content-Encoding).
+func hasStreamingPayloadToken(vals []string) bool {
+	for _, tok := range flattenHeaderTokens(vals) {
+		if strings.HasPrefix(strings.ToLower(tok), "streaming-") {
+			return true
+		}
+	}
+	return false
+}
+
+// shouldDecodeAwsChunked reports whether the request asks for aws-chunked
+// decoding: an aws-chunked Content-Encoding token, or a STREAMING-*
+// content hash without Content-Encoding. Uses Header.Values so split
+// multi-value headers are all visible (headerMap keeps first-only).
+func shouldDecodeAwsChunked(h http.Header) bool {
+	return hasAwsChunkedToken(h.Values("Content-Encoding")) ||
+		hasStreamingPayloadToken(h.Values("X-Amz-Content-Sha256"))
+}
+
+// parseDecodedContentLength flattens x-amz-decoded-content-length values:
+// exactly one valid decimal token verifies against the decoded length,
+// zero/multiple/invalid tokens are ignored.
+func parseDecodedContentLength(vals []string) (uint64, bool) {
+	toks := flattenHeaderTokens(vals)
+	if len(toks) != 1 {
+		return 0, false
+	}
+	s := toks[0]
+	for i := 0; i < len(s); i++ {
+		if s[i] < '0' || s[i] > '9' {
+			return 0, false
+		}
+	}
+	n, err := strconv.ParseUint(s, 10, 64)
+	if err != nil {
+		return 0, false
+	}
+	return n, true
+}
+
+// trimOWS trims spaces and horizontal tabs (HTTP optional whitespace).
+func trimOWS(b []byte) []byte {
+	return bytes.Trim(b, " \t")
+}
+
+// decodeAwsChunked strips SigV4 STREAMING framing from encoded, returning
+// the decoded bytes. bodyLimit bounds the decoded total (exceeding it is
+// a 413); every other framing violation is a 400 IncompleteBody.
+//
+// Grammar: strict CRLF (bare LF or lone CR fail); chunk sizes are
+// OWS-trimmed hex (uppercase ok) with explicit rejection of +/- signs and
+// 0x prefixes before ParseUint(16,64); everything after the first ';' is
+// an opaque extension (bare sizes without extensions are accepted in both
+// streaming variants). Chunk data is exactly size bytes plus CRLF. The
+// terminal zero chunk may carry trailer lines (Name: value, split at the
+// first colon, empty value ok) which are discarded, never merged into
+// request headers; checksum trailers are discarded the same way. Caps:
+// lines <= 4KB, data chunks <= 10k, trailers <= 32 lines / 8KB total,
+// per-chunk and running decoded totals <= bodyLimit (incremental).
+func decodeAwsChunked(encoded []byte, bodyLimit int64, decodedLengthVals []string) ([]byte, error) {
+	fail := &awsChunkedError{status: http.StatusBadRequest}
+	tooBig := &awsChunkedError{status: http.StatusRequestEntityTooLarge}
+
+	var decoded []byte
+	chunks := 0
+	pos := 0
+	// readLine consumes one strict-CRLF line (returned without the CRLF).
+	readLine := func() ([]byte, error) {
+		end := -1
+		for i := pos; i < len(encoded); i++ {
+			if encoded[i] == '\n' {
+				end = i
+				break
+			}
+		}
+		if end < 0 {
+			return nil, fail // unterminated
+		}
+		if end == pos || encoded[end-1] != '\r' {
+			return nil, fail // bare LF
+		}
+		line := encoded[pos : end-1]
+		if bytes.IndexByte(line, '\r') >= 0 {
+			return nil, fail // lone CR
+		}
+		if end+1-pos > awsChunkedMaxLine {
+			return nil, fail
+		}
+		pos = end + 1
+		return line, nil
+	}
+
+	for {
+		line, err := readLine()
+		if err != nil {
+			return nil, err
+		}
+		sizeField := line
+		if i := bytes.IndexByte(line, ';'); i >= 0 {
+			sizeField = line[:i]
+		}
+		sizeStr := trimOWS(sizeField)
+		if len(sizeStr) == 0 {
+			return nil, fail
+		}
+		if sizeStr[0] == '+' || sizeStr[0] == '-' {
+			return nil, fail
+		}
+		if len(sizeStr) >= 2 && sizeStr[0] == '0' && (sizeStr[1] == 'x' || sizeStr[1] == 'X') {
+			return nil, fail
+		}
+		size, err := strconv.ParseUint(string(sizeStr), 16, 64)
+		if err != nil {
+			return nil, fail
+		}
+		if size == 0 {
+			trailerBytes := 0
+			for n := 0; ; n++ {
+				tline, err := readLine()
+				if err != nil {
+					return nil, err
+				}
+				if len(tline) == 0 {
+					break // terminal empty line
+				}
+				if n+1 > awsChunkedMaxTrailers {
+					return nil, fail
+				}
+				trailerBytes += len(tline) + 2 // raw line incl CRLF
+				if trailerBytes > awsChunkedMaxTrailerBytes {
+					return nil, fail
+				}
+				ci := bytes.IndexByte(tline, ':')
+				if ci < 0 {
+					return nil, fail // no colon
+				}
+				if len(trimOWS(tline[:ci])) == 0 {
+					return nil, fail // empty name
+				}
+				// Value (possibly empty) is OWS-trimmed, then the
+				// trailer is discarded — never merged into headers.
+			}
+			if pos != len(encoded) {
+				return nil, fail // trailing garbage
+			}
+			break
+		}
+		// Incremental decoded bound before touching the data.
+		chunks++
+		if chunks > awsChunkedMaxChunks {
+			return nil, fail
+		}
+		if bodyLimit < 0 || size > uint64(bodyLimit) || uint64(len(decoded)) > uint64(bodyLimit)-size {
+			return nil, tooBig
+		}
+		remaining := uint64(len(encoded) - pos)
+		if size+2 < size || size+2 > remaining {
+			return nil, fail // truncated data
+		}
+		if encoded[pos+int(size)] != '\r' || encoded[pos+int(size)+1] != '\n' {
+			return nil, fail
+		}
+		decoded = append(decoded, encoded[pos:pos+int(size)]...)
+		pos += int(size) + 2
+	}
+
+	if want, ok := parseDecodedContentLength(decodedLengthVals); ok {
+		if uint64(len(decoded)) != want {
+			return nil, fail
+		}
+	}
+	return decoded, nil
+}