diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b05537..5061902 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,11 @@ All notable changes to **stunt** are documented here. The format is based on `If-None-Match`, `If-Modified-Since`, and `If-Unmodified-Since`, returning `304` or `412`; `PUT`/`DELETE` honor the ETag conditions. +### Engine + +- **aws-chunked request bodies are decoded before dispatch.** SigV4 streaming + uploads from AWS SDKs store the object bytes instead of the chunk framing. + ## [0.52.0] — 2026-08-24 The conformance campaign: every real API adapter now carries a real test diff --git a/CONFORMANCE.md b/CONFORMANCE.md index 573001d..5138b66 100644 --- a/CONFORMANCE.md +++ b/CONFORMANCE.md @@ -40,7 +40,7 @@ Behavior columns come in two kinds: **verified** (an official SDK was driven aga | [avalara-style](adapters/avalara-style/) | Avalara AvaTax REST API `2` | 8 | VM | — | — | [5](#avalara-style) | [3](#avalara-style) | | [aws-cognito-style](adapters/aws-cognito-style/) | Amazon Cognito Identity Provider API `2016-04-18` | 7 | VM | — | — | [6](#aws-cognito-style) | [3](#aws-cognito-style) | | [aws-iam-sts-style](adapters/aws-iam-sts-style/) | AWS STS + IAM API `2011-06-15` | 2 | SDK | aws-sdk-go-v2 @ v1.43.7 | 2 | [3](#aws-iam-sts-style) | [3](#aws-iam-sts-style) | -| [aws-s3-style](adapters/aws-s3-style/) | Amazon S3 API `2006-03-01` | 8 | SDK | aws-sdk-go-v2 @ v1.43.7 | 6 | [5](#aws-s3-style) | [7](#aws-s3-style) | +| [aws-s3-style](adapters/aws-s3-style/) | Amazon S3 API `2006-03-01` | 8 | SDK | aws-sdk-go-v2 @ v1.43.7 | 6 | [5](#aws-s3-style) | [9](#aws-s3-style) | | [azure-devops-style](adapters/azure-devops-style/) | Azure DevOps REST API `7.1` | 17 | VM | — | — | [8](#azure-devops-style) | [6](#azure-devops-style) | | [azure-servicebus-style](adapters/azure-servicebus-style/) | Azure Service Bus + Storage `2024-01-01` | 18 | VM | — | — | [6](#azure-servicebus-style) | [3](#azure-servicebus-style) | | [azure-storage-style](adapters/azure-storage-style/) | Azure Storage Blob REST API `2024-08-04` | 9 | VM | — | — | [6](#azure-storage-style) | [3](#azure-storage-style) | @@ -2598,11 +2598,13 @@ behavior notes live in each adapter's README. - No browser form POST uploads (POST policy) - No ListMultipartUploads (GET /{bucket}?uploads) -**Deviations** (7) +**Deviations** (9) - ETags are MD5 hex (multipart MD5(binary-concat)-N) - Multipart 5 MiB minimum part size not enforced (small parts allowed) - DELETE of a missing bucket is an idempotent 204 (real S3: 404 NoSuchBucket) +- Per-chunk STREAMING signatures not verified (header SigV4 only) +- Streaming checksum trailers discarded (unsupported-checksum) - DELETE object against a missing bucket is 204 (real S3: 404 NoSuchBucket) - GET/HEAD object against a missing bucket is 404 NoSuchKey (real S3: 404 NoSuchBucket) - SigV4 canonical URI/query rebuilt from decoded values — duplicates indistinguishable diff --git a/adapters/aws-s3-style/README.md b/adapters/aws-s3-style/README.md index 202f3e3..85a12a2 100644 --- a/adapters/aws-s3-style/README.md +++ b/adapters/aws-s3-style/README.md @@ -123,6 +123,24 @@ Two documented deviations: weak validators (`W/"..."`) compare as strong, and a `DELETE` or `GET`/`HEAD` against a missing bucket is `204`/`NoSuchKey` where real S3 returns `NoSuchBucket`. +### Streaming uploads (aws-chunked) + +A default SDK `PutObject` signs its payload with +`STREAMING-AWS4-HMAC-SHA256-PAYLOAD` and wraps the bytes in +`Content-Encoding: aws-chunked` framing. The engine decodes that framing +before rules, profiles, and handlers run, so the adapter stores the object +bytes. `Content-Encoding` and `x-amz-content-sha256` pass through untouched, +since SigV4 signs them. + +Framing that is malformed, truncated, or over the limits below returns +`400 IncompleteBody`; decoded output over `max_body_bytes` returns `413`. +Limits are 4 KiB per chunk or trailer line, 10,000 data chunks, and 32 +trailers totalling 8 KiB. `x-amz-decoded-content-length` is verified when +exactly one valid value is present and ignored otherwise. + +Two documented deviations: per-chunk signatures are not verified (the +header signature is), and streaming checksum trailers are discarded. + ## Auth — AWS Signature Version 4 (SigV4), verified for real Amazon S3 uses **AWS Signature Version 4** (SigV4) for authentication. This @@ -273,6 +291,7 @@ All errors use S3-shaped XML: | `InvalidPart` | 400 | CompleteMultipartUpload lists a part that was never uploaded, or whose ETag does not match | | `InvalidPartOrder` | 400 | CompleteMultipartUpload part list is not in ascending order | | `MalformedXML` | 400 | CompleteMultipartUpload body is not valid `CompleteMultipartUpload` XML | +| `IncompleteBody` | 400 | `aws-chunked` framing is malformed, truncated, or over the decoder limits | | `PreconditionFailed` | 412 | An `If-Match`/`If-None-Match`/`If-Unmodified-Since`/`If-Modified-Since` condition did not hold | | `MethodNotAllowed` | 405 | POST to an object without `?uploads`/`?uploadId` | | `NoSuchBucket` | 404 | Bucket doesn't exist | diff --git a/conformance/matrix.json b/conformance/matrix.json index fe3ce90..4815518 100644 --- a/conformance/matrix.json +++ b/conformance/matrix.json @@ -9383,6 +9383,8 @@ "ETags are MD5 hex (multipart MD5(binary-concat)-N)", "Multipart 5 MiB minimum part size not enforced (small parts allowed)", "DELETE of a missing bucket is an idempotent 204 (real S3: 404 NoSuchBucket)", + "Per-chunk STREAMING signatures not verified (header SigV4 only)", + "Streaming checksum trailers discarded (unsupported-checksum)", "DELETE object against a missing bucket is 204 (real S3: 404 NoSuchBucket)", "GET/HEAD object against a missing bucket is 404 NoSuchKey (real S3: 404 NoSuchBucket)", "SigV4 canonical URI/query rebuilt from decoded values — duplicates indistinguishable", diff --git a/conformance/matrix.yaml b/conformance/matrix.yaml index 98e5f23..754291c 100644 --- a/conformance/matrix.yaml +++ b/conformance/matrix.yaml @@ -152,6 +152,8 @@ adapters: - "ETags are MD5 hex (multipart MD5(binary-concat)-N)" - "Multipart 5 MiB minimum part size not enforced (small parts allowed)" - "DELETE of a missing bucket is an idempotent 204 (real S3: 404 NoSuchBucket)" + - "Per-chunk STREAMING signatures not verified (header SigV4 only)" + - "Streaming checksum trailers discarded (unsupported-checksum)" - "DELETE object against a missing bucket is 204 (real S3: 404 NoSuchBucket)" - "GET/HEAD object against a missing bucket is 404 NoSuchKey (real S3: 404 NoSuchBucket)" - "SigV4 canonical URI/query rebuilt from decoded values — duplicates indistinguishable" diff --git a/internal/engine/aws_s3_style_test.go b/internal/engine/aws_s3_style_test.go index 7fede88..9bbdc9c 100644 --- a/internal/engine/aws_s3_style_test.go +++ b/internal/engine/aws_s3_style_test.go @@ -96,9 +96,15 @@ func awsSigV4SigningKey(secret, date, region, service string) []byte { // x-amz-date are signed and the payload hash covers the body bytes. func awsSigV4Sign(t *testing.T, req *http.Request, body []byte, service, accessKey, secretKey string, at time.Time) { t.Helper() + awsSigV4SignPayload(t, req, awsSHA256Hex(body), service, accessKey, secretKey, at) +} + +// awsSigV4SignPayload signs req in place with real SigV4 using an explicit +// payload hash. Streaming (aws-chunked) uploads sign the STREAMING literal +// rather than the body bytes, so chunked tests pass the literal here. +func awsSigV4SignPayload(t *testing.T, req *http.Request, payloadHash, service, accessKey, secretKey string, at time.Time) { amzDate := at.UTC().Format("20060102T150405Z") date := amzDate[:8] - payloadHash := awsSHA256Hex(body) req.Header.Set("x-amz-date", amzDate) signedHeaders := []string{"host", "x-amz-date"} diff --git a/internal/engine/chunked_aws.go b/internal/engine/chunked_aws.go new file mode 100644 index 0000000..2263bea --- /dev/null +++ b/internal/engine/chunked_aws.go @@ -0,0 +1,255 @@ +// AWS-chunked (SigV4 STREAMING) framing decode. +// +// S3 SDKs using SigV4 streaming uploads send the object bytes wrapped in +// aws-chunked framing: hex-size chunk lines with opaque ";extensions", +// strict CRLF delimiters, a terminal zero chunk, and optional trailers. +// The engine strips that framing before adapter dispatch so handlers see +// the decoded bytes, while the Content-Encoding and x-amz-content-sha256 +// headers are preserved untouched (SigV4 signs them). +package engine + +import ( + "bytes" + "net/http" + "strconv" + "strings" +) + +const ( + // awsChunkedReadOverhead is the headroom above bodyLimit granted to + // the encoded (still framed) read: framing, extensions, and trailers + // for up to awsChunkedMaxChunks chunks. + awsChunkedReadOverhead = int64(2 << 20) // 2 MiB + // awsChunkedMaxLine caps one chunk-size or trailer line (incl CRLF). + awsChunkedMaxLine = 4096 + // awsChunkedMaxChunks caps data chunks per request. + awsChunkedMaxChunks = 10000 + // awsChunkedMaxTrailers caps trailer header lines after the zero chunk. + awsChunkedMaxTrailers = 32 + // awsChunkedMaxTrailerBytes caps total trailer line bytes (incl CRLF). + awsChunkedMaxTrailerBytes = 8 << 10 +) + +// awsChunkedError is a decode failure: status is 400 (framing) or 413 +// (decoded/encoded over limit). The 400 body carries the S3 IncompleteBody +// code; the error is never echoed back with request bytes. +type awsChunkedError struct { + status int +} + +func (e *awsChunkedError) Error() string { + if e.status == http.StatusRequestEntityTooLarge { + return "aws-chunked decoded body exceeds limit" + } + return "IncompleteBody" +} + +// writeIncompleteBody writes the S3 IncompleteBody error envelope for +// aws-chunked framing failures. The request bytes are never echoed. +func writeIncompleteBody(w http.ResponseWriter) { + w.Header().Set("Content-Type", "application/xml") + w.WriteHeader(http.StatusBadRequest) + _, _ = w.Write([]byte(`IncompleteBodyThe request body terminated unexpectedly or did not match the expected length.`)) +} + +// flattenHeaderTokens flattens multi-value headers into comma-split, +// OWS-trimmed tokens (empty members dropped per RFC 9110 list parsing). +func flattenHeaderTokens(vals []string) []string { + var out []string + for _, v := range vals { + for _, tok := range strings.Split(v, ",") { + tok = strings.Trim(tok, " \t") + if tok == "" { + continue + } + out = append(out, tok) + } + } + return out +} + +// hasAwsChunkedToken reports whether any Content-Encoding value carries +// the aws-chunked token (comma-split, OWS-trimmed, case-folded). +func hasAwsChunkedToken(vals []string) bool { + for _, tok := range flattenHeaderTokens(vals) { + if strings.ToLower(tok) == "aws-chunked" { + return true + } + } + return false +} + +// hasStreamingPayloadToken reports whether any x-amz-content-sha256 value +// is a STREAMING-* token (malformed-client trigger without Content-Encoding). +func hasStreamingPayloadToken(vals []string) bool { + for _, tok := range flattenHeaderTokens(vals) { + if strings.HasPrefix(strings.ToLower(tok), "streaming-") { + return true + } + } + return false +} + +// shouldDecodeAwsChunked reports whether the request asks for aws-chunked +// decoding: an aws-chunked Content-Encoding token, or a STREAMING-* +// content hash without Content-Encoding. Uses Header.Values so split +// multi-value headers are all visible (headerMap keeps first-only). +func shouldDecodeAwsChunked(h http.Header) bool { + return hasAwsChunkedToken(h.Values("Content-Encoding")) || + hasStreamingPayloadToken(h.Values("X-Amz-Content-Sha256")) +} + +// parseDecodedContentLength flattens x-amz-decoded-content-length values: +// exactly one valid decimal token verifies against the decoded length, +// zero/multiple/invalid tokens are ignored. +func parseDecodedContentLength(vals []string) (uint64, bool) { + toks := flattenHeaderTokens(vals) + if len(toks) != 1 { + return 0, false + } + s := toks[0] + for i := 0; i < len(s); i++ { + if s[i] < '0' || s[i] > '9' { + return 0, false + } + } + n, err := strconv.ParseUint(s, 10, 64) + if err != nil { + return 0, false + } + return n, true +} + +// trimOWS trims spaces and horizontal tabs (HTTP optional whitespace). +func trimOWS(b []byte) []byte { + return bytes.Trim(b, " \t") +} + +// decodeAwsChunked strips SigV4 STREAMING framing from encoded, returning +// the decoded bytes. bodyLimit bounds the decoded total (exceeding it is +// a 413); every other framing violation is a 400 IncompleteBody. +// +// Grammar: strict CRLF (bare LF or lone CR fail); chunk sizes are +// OWS-trimmed hex (uppercase ok) with explicit rejection of +/- signs and +// 0x prefixes before ParseUint(16,64); everything after the first ';' is +// an opaque extension (bare sizes without extensions are accepted in both +// streaming variants). Chunk data is exactly size bytes plus CRLF. The +// terminal zero chunk may carry trailer lines (Name: value, split at the +// first colon, empty value ok) which are discarded, never merged into +// request headers; checksum trailers are discarded the same way. Caps: +// lines <= 4KB, data chunks <= 10k, trailers <= 32 lines / 8KB total, +// per-chunk and running decoded totals <= bodyLimit (incremental). +func decodeAwsChunked(encoded []byte, bodyLimit int64, decodedLengthVals []string) ([]byte, error) { + fail := &awsChunkedError{status: http.StatusBadRequest} + tooBig := &awsChunkedError{status: http.StatusRequestEntityTooLarge} + + var decoded []byte + chunks := 0 + pos := 0 + // readLine consumes one strict-CRLF line (returned without the CRLF). + readLine := func() ([]byte, error) { + end := -1 + for i := pos; i < len(encoded); i++ { + if encoded[i] == '\n' { + end = i + break + } + } + if end < 0 { + return nil, fail // unterminated + } + if end == pos || encoded[end-1] != '\r' { + return nil, fail // bare LF + } + line := encoded[pos : end-1] + if bytes.IndexByte(line, '\r') >= 0 { + return nil, fail // lone CR + } + if end+1-pos > awsChunkedMaxLine { + return nil, fail + } + pos = end + 1 + return line, nil + } + + for { + line, err := readLine() + if err != nil { + return nil, err + } + sizeField := line + if i := bytes.IndexByte(line, ';'); i >= 0 { + sizeField = line[:i] + } + sizeStr := trimOWS(sizeField) + if len(sizeStr) == 0 { + return nil, fail + } + if sizeStr[0] == '+' || sizeStr[0] == '-' { + return nil, fail + } + if len(sizeStr) >= 2 && sizeStr[0] == '0' && (sizeStr[1] == 'x' || sizeStr[1] == 'X') { + return nil, fail + } + size, err := strconv.ParseUint(string(sizeStr), 16, 64) + if err != nil { + return nil, fail + } + if size == 0 { + trailerBytes := 0 + for n := 0; ; n++ { + tline, err := readLine() + if err != nil { + return nil, err + } + if len(tline) == 0 { + break // terminal empty line + } + if n+1 > awsChunkedMaxTrailers { + return nil, fail + } + trailerBytes += len(tline) + 2 // raw line incl CRLF + if trailerBytes > awsChunkedMaxTrailerBytes { + return nil, fail + } + ci := bytes.IndexByte(tline, ':') + if ci < 0 { + return nil, fail // no colon + } + if len(trimOWS(tline[:ci])) == 0 { + return nil, fail // empty name + } + // Value (possibly empty) is OWS-trimmed, then the + // trailer is discarded — never merged into headers. + } + if pos != len(encoded) { + return nil, fail // trailing garbage + } + break + } + // Incremental decoded bound before touching the data. + chunks++ + if chunks > awsChunkedMaxChunks { + return nil, fail + } + if bodyLimit < 0 || size > uint64(bodyLimit) || uint64(len(decoded)) > uint64(bodyLimit)-size { + return nil, tooBig + } + remaining := uint64(len(encoded) - pos) + if size+2 < size || size+2 > remaining { + return nil, fail // truncated data + } + if encoded[pos+int(size)] != '\r' || encoded[pos+int(size)+1] != '\n' { + return nil, fail + } + decoded = append(decoded, encoded[pos:pos+int(size)]...) + pos += int(size) + 2 + } + + if want, ok := parseDecodedContentLength(decodedLengthVals); ok { + if uint64(len(decoded)) != want { + return nil, fail + } + } + return decoded, nil +} diff --git a/internal/engine/chunked_aws_test.go b/internal/engine/chunked_aws_test.go new file mode 100644 index 0000000..2a2f3a6 --- /dev/null +++ b/internal/engine/chunked_aws_test.go @@ -0,0 +1,394 @@ +package engine + +import ( + "bytes" + "context" + "fmt" + "io" + "net/http" + "path/filepath" + "strconv" + "strings" + "testing" + "time" + + "stuntapi.com/stunt/internal/manifest" +) + +// awsChunkedFraming wraps data in SigV4 STREAMING (aws-chunked) framing: +// one data chunk plus the terminating zero chunk, with opaque +// chunk-signature extensions (per-chunk signatures are not verified, +// header SigV4 only). +func awsChunkedFraming(data []byte) []byte { + sig := strings.Repeat("a", 64) + var b strings.Builder + fmt.Fprintf(&b, "%x;chunk-signature=%s\r\n", len(data), sig) + b.Write(data) + b.WriteString("\r\n") + fmt.Fprintf(&b, "0;chunk-signature=%s\r\n\r\n", sig) + return []byte(b.String()) +} + +// s3SignedReqChunked builds a SigV4-signed PUT carrying aws-chunked +// framing, as real S3 SDKs do: Content-Encoding: aws-chunked, the +// x-amz-content-sha256 header carries the STREAMING literal, and the +// signature covers that literal (not the body bytes). +func s3SignedReqChunked(t *testing.T, rawurl string, framing []byte, decodedLen int, at time.Time) *http.Request { + t.Helper() + req, err := http.NewRequest("PUT", rawurl, bytes.NewReader(framing)) + if err != nil { + t.Fatal(err) + } + req.Header.Set("Content-Type", "application/octet-stream") + req.Header.Set("Content-Encoding", "aws-chunked") + req.Header.Set("x-amz-decoded-content-length", strconv.Itoa(decodedLen)) + const streamHash = "STREAMING-AWS4-HMAC-SHA256-PAYLOAD" + req.Header.Set("x-amz-content-sha256", streamHash) + awsSigV4SignPayload(t, req, streamHash, "s3", awsStyleAccessKey, awsStyleSecretKey, at) + return req +} + +// TestAWSChunkedDecode is the S3 5B→5B round-trip: a PUT carrying +// aws-chunked framing must be stored decoded, so GET returns the 5 +// decoded bytes (not the framing) and HEAD reports ContentLength 5. +func TestAWSChunkedDecode(t *testing.T) { + adapterDir, err := filepath.Abs(filepath.Join("..", "..", "adapters", "aws-s3-style")) + if err != nil { + t.Fatal(err) + } + stateDir := t.TempDir() + m := &manifest.Manifest{ + Path: filepath.Join(stateDir, "stunt.yaml"), + Version: 1, + Network: manifest.Network{Mode: "port", BasePort: 0}, + Services: map[string]manifest.Service{ + "s3": {Adapter: adapterDir}, + }, + } + + e, err := New(m) + if err != nil { + t.Fatalf("engine.New: %v", err) + } + defer e.Close() + + addrs, cancel, err := e.ServeForTest(context.Background()) + if err != nil { + t.Fatalf("ServeForTest: %v", err) + } + defer cancel() + time.Sleep(50 * time.Millisecond) + + base := addrs["s3"] + now := time.Now() + + if _, status := s3Put(t, base+"/chunkbucket", nil, now); status != 200 { + t.Fatalf("create bucket -> status %d, want 200", status) + } + + framing := awsChunkedFraming([]byte("hello")) + + resp, err := http.DefaultClient.Do(s3SignedReqChunked(t, base+"/chunkbucket/hello.txt", framing, 5, now)) + if err != nil { + t.Fatal(err) + } + putBody, _ := io.ReadAll(resp.Body) + resp.Body.Close() + if resp.StatusCode != 200 { + t.Fatalf("chunked put -> status %d, want 200; body %s", resp.StatusCode, putBody) + } + // The ETag digest algorithm is covered separately; here what matters is + // that it is derived from the decoded bytes, not the framing. A quoted + // 32/64-char hex digest is the shape every real S3 ETag has. + if etag := resp.Header.Get("ETag"); len(etag) < 34 || etag[0] != '"' || etag[len(etag)-1] != '"' { + t.Fatalf("chunked put ETag = %q, want a quoted content digest", etag) + } + + body, status := s3Get(t, base+"/chunkbucket/hello.txt", now) + if status != 200 { + t.Fatalf("get chunked object -> status %d, want 200; body %s", status, body) + } + if body != "hello" { + t.Fatalf("chunked round-trip body = %q (%dB), want %q (5B)", body, len(body), "hello") + } + + hresp := s3Head(t, base+"/chunkbucket/hello.txt", now) + hresp.Body.Close() + if hresp.StatusCode != 200 { + t.Fatalf("head chunked object -> status %d, want 200", hresp.StatusCode) + } + if cl := hresp.Header.Get("Content-Length"); cl != "5" { + t.Fatalf("rawHttpBodyLength %s, want 5", cl) + } + + // Declared decoded length mismatch -> 400 IncompleteBody. + resp, err = http.DefaultClient.Do(s3SignedReqChunked(t, base+"/chunkbucket/bad.txt", framing, 6, now)) + if err != nil { + t.Fatal(err) + } + mismatchBody, _ := io.ReadAll(resp.Body) + resp.Body.Close() + if resp.StatusCode != 400 { + t.Fatalf("decoded-length mismatch -> status %d, want 400; body %s", resp.StatusCode, mismatchBody) + } + if !strings.Contains(string(mismatchBody), "IncompleteBody") { + t.Fatalf("decoded-length mismatch: missing IncompleteBody; body %s", mismatchBody) + } + + // STREAMING sha256 without Content-Encoding and without framing + // (malformed client) -> decode attempt -> 400 IncompleteBody. + plain := []byte("hello") + req, err := http.NewRequest("PUT", base+"/chunkbucket/plain.txt", bytes.NewReader(plain)) + if err != nil { + t.Fatal(err) + } + req.Header.Set("Content-Type", "application/octet-stream") + const streamHash = "STREAMING-AWS4-HMAC-SHA256-PAYLOAD" + req.Header.Set("x-amz-content-sha256", streamHash) + awsSigV4SignPayload(t, req, streamHash, "s3", awsStyleAccessKey, awsStyleSecretKey, now) + resp, err = http.DefaultClient.Do(req) + if err != nil { + t.Fatal(err) + } + plainBody, _ := io.ReadAll(resp.Body) + resp.Body.Close() + if resp.StatusCode != 400 { + t.Fatalf("streaming-without-framing -> status %d, want 400; body %s", resp.StatusCode, plainBody) + } + if !strings.Contains(string(plainBody), "IncompleteBody") { + t.Fatalf("streaming-without-framing: missing IncompleteBody; body %s", plainBody) + } +} + +// chunkedStatus decodes enc and returns the resulting status: 200 on +// success, else the aws-chunked error status (400 framing, 413 over +// limit). +func chunkedStatus(t *testing.T, enc []byte, bodyLimit int64, decLenVals []string) (int, []byte) { + t.Helper() + dec, err := decodeAwsChunked(enc, bodyLimit, decLenVals) + if err == nil { + return 200, dec + } + cerr, ok := err.(*awsChunkedError) + if !ok { + t.Fatalf("decode error type = %T, want *awsChunkedError", err) + } + return cerr.status, nil +} + +func TestAWSChunkedParser(t *testing.T) { + const limit = int64(1 << 20) + + t.Run("single chunk with extension", func(t *testing.T) { + st, dec := chunkedStatus(t, []byte("5;chunk-signature=abc\r\nhello\r\n0;chunk-signature=abc\r\n\r\n"), limit, nil) + if st != 200 || string(dec) != "hello" { + t.Fatalf("status=%d body=%q, want 200 hello", st, dec) + } + }) + t.Run("multi chunk bare sizes", func(t *testing.T) { + enc := []byte("5\r\nhello\r\n1\r\n \r\n1\r\n!\r\n0\r\n\r\n") + st, dec := chunkedStatus(t, enc, limit, nil) + if st != 200 || string(dec) != "hello !" { + t.Fatalf("status=%d body=%q, want 200 'hello !'", st, dec) + } + }) + t.Run("uppercase hex ok", func(t *testing.T) { + st, dec := chunkedStatus(t, []byte("A\r\n0123456789\r\n0\r\n\r\n"), limit, nil) + if st != 200 || string(dec) != "0123456789" { + t.Fatalf("status=%d body=%q, want 200", st, dec) + } + }) + t.Run("ows trim tab and space before semicolon", func(t *testing.T) { + st, dec := chunkedStatus(t, []byte(" \t5 \t;chunk-signature=x\r\nhello\r\n0\r\n\r\n"), limit, nil) + if st != 200 || string(dec) != "hello" { + t.Fatalf("status=%d body=%q, want 200 hello", st, dec) + } + }) + t.Run("sign and hex prefix rejected", func(t *testing.T) { + for _, enc := range []string{ + "+5\r\nhello\r\n0\r\n\r\n", + "-5\r\nhello\r\n0\r\n\r\n", + "0x5\r\nhello\r\n0\r\n\r\n", + "0X5\r\nhello\r\n0\r\n\r\n", + "\r\nhello\r\n0\r\n\r\n", + "zz\r\nhello\r\n0\r\n\r\n", + } { + if st, _ := chunkedStatus(t, []byte(enc), limit, nil); st != 400 { + t.Fatalf("%q -> status %d, want 400", enc, st) + } + } + }) + t.Run("bare LF and lone CR rejected", func(t *testing.T) { + for _, enc := range []string{ + "5;chunk-signature=abc\nhello\r\n0\r\n\r\n", // bare LF ends size line + "5;chunk-signature=a\rc\r\nhello\r\n0\r\n\r\n", // lone CR in size line + "5\r\nhello\r\n0\r\n\n", // bare LF ends final empty line + } { + if st, _ := chunkedStatus(t, []byte(enc), limit, nil); st != 400 { + t.Fatalf("%q -> status %d, want 400", enc, st) + } + } + }) + t.Run("truncated and short data rejected", func(t *testing.T) { + for _, enc := range []string{ + "5\r\nhello", // truncated mid-data + "5\r\nhell\r\n0\r\n\r\n", // data short, CRLF elsewhere + "5\r\nhelloXX0\r\n\r\n", // missing CRLF after data + "5\r\nhello\r\n", // missing final chunk + } { + if st, _ := chunkedStatus(t, []byte(enc), limit, nil); st != 400 { + t.Fatalf("%q -> status %d, want 400", enc, st) + } + } + }) + t.Run("zero chunk yields empty", func(t *testing.T) { + st, dec := chunkedStatus(t, []byte("0\r\n\r\n"), limit, nil) + if st != 200 || len(dec) != 0 { + t.Fatalf("status=%d body=%q, want 200 empty", st, dec) + } + }) + t.Run("trailers discarded never merged", func(t *testing.T) { + enc := []byte("5\r\nhello\r\n0\r\nX-Amz-Checksum-Crc32c: abc:def\r\nX-Empty:\r\n\r\n") + st, dec := chunkedStatus(t, enc, limit, nil) + if st != 200 || string(dec) != "hello" { + t.Fatalf("status=%d body=%q, want 200 hello", st, dec) + } + }) + t.Run("trailer malformed rejected", func(t *testing.T) { + for _, enc := range []string{ + "0\r\n: novalue\r\n\r\n", // empty name + "0\r\nno-colon\r\n\r\n", // no colon + "0\r\n \r\n\r\n", // whitespace-only line (no colon) + } { + if st, _ := chunkedStatus(t, []byte(enc), limit, nil); st != 400 { + t.Fatalf("%q -> status %d, want 400", enc, st) + } + } + }) + t.Run("trailing garbage rejected", func(t *testing.T) { + if st, _ := chunkedStatus(t, []byte("0\r\n\r\nzzz"), limit, nil); st != 400 { + t.Fatalf("trailing garbage -> status %d, want 400", st) + } + }) + t.Run("decoded length verify", func(t *testing.T) { + enc := []byte("5\r\nhello\r\n0\r\n\r\n") + if st, _ := chunkedStatus(t, enc, limit, []string{"5"}); st != 200 { + t.Fatalf("exact length -> status %d, want 200", st) + } + if st, _ := chunkedStatus(t, enc, limit, []string{" \t5\t "}); st != 200 { + t.Fatalf("ows-padded length -> status %d, want 200", st) + } + if st, _ := chunkedStatus(t, enc, limit, []string{"6"}); st != 400 { + t.Fatalf("mismatch -> status %d, want 400", st) + } + if st, _ := chunkedStatus(t, enc, limit, []string{"4"}); st != 400 { + t.Fatalf("short mismatch -> status %d, want 400", st) + } + // Zero/multiple/invalid are ignored: decode succeeds. + for _, vals := range [][]string{nil, {}, {""}, {"abc"}, {"5", "5"}, {"5,6"}, {"99999999999999999999999"}} { + if st, _ := chunkedStatus(t, enc, limit, vals); st != 200 { + t.Fatalf("vals %q -> status %d, want 200 (ignored)", vals, st) + } + } + }) + t.Run("decoded over body limit", func(t *testing.T) { + // Single chunk larger than the limit. + if st, _ := chunkedStatus(t, []byte("5\r\nhello\r\n0\r\n\r\n"), 4, nil); st != 413 { + t.Fatalf("single over limit -> status %d, want 413", st) + } + // Running total crosses the limit on the second chunk. + enc := []byte("3\r\nabc\r\n3\r\ndef\r\n0\r\n\r\n") + if st, _ := chunkedStatus(t, enc, 5, nil); st != 413 { + t.Fatalf("running over limit -> status %d, want 413", st) + } + if st, _ := chunkedStatus(t, enc, 6, nil); st != 200 { + t.Fatalf("exact limit -> status %d, want 200", st) + } + }) + t.Run("chunk count cap", func(t *testing.T) { + var ok strings.Builder + for i := 0; i < awsChunkedMaxChunks; i++ { + ok.WriteString("1\r\na\r\n") + } + ok.WriteString("0\r\n\r\n") + if st, dec := chunkedStatus(t, []byte(ok.String()), 1<<20, nil); st != 200 || len(dec) != awsChunkedMaxChunks { + t.Fatalf("10000 chunks -> status %d len %d, want 200/%d", st, len(dec), awsChunkedMaxChunks) + } + var over strings.Builder + for i := 0; i <= awsChunkedMaxChunks; i++ { + over.WriteString("1\r\na\r\n") + } + over.WriteString("0\r\n\r\n") + if st, _ := chunkedStatus(t, []byte(over.String()), 1<<20, nil); st != 400 { + t.Fatalf("10001 chunks -> status %d, want 400", st) + } + }) + t.Run("trailer caps", func(t *testing.T) { + var many strings.Builder + many.WriteString("0\r\n") + for i := 0; i <= awsChunkedMaxTrailers; i++ { + fmt.Fprintf(&many, "X-T-%d: v\r\n", i) + } + many.WriteString("\r\n") + if st, _ := chunkedStatus(t, []byte(many.String()), limit, nil); st != 400 { + t.Fatalf("33 trailers -> status %d, want 400", st) + } + big := "0\r\nX-Big: " + strings.Repeat("v", 8<<10) + "\r\n\r\n" + if st, _ := chunkedStatus(t, []byte(big), limit, nil); st != 400 { + t.Fatalf("8KB trailer -> status %d, want 400", st) + } + long := strings.Repeat("x", 5000) + "\r\n" + if st, _ := chunkedStatus(t, []byte(long), limit, nil); st != 400 { + t.Fatalf("5KB size line -> status %d, want 400", st) + } + }) +} + +func TestAWSChunkedTrigger(t *testing.T) { + ce := func(vals ...string) http.Header { + h := http.Header{} + for _, v := range vals { + h.Add("Content-Encoding", v) + } + return h + } + cases := []struct { + header http.Header + want bool + }{ + {ce("aws-chunked"), true}, + {ce("AWS-Chunked"), true}, + {ce("gzip, Aws-Chunked"), true}, + {ce("aws-chunked", "gzip"), true}, // split multi-value header + {ce("aws-chunked, aws-chunked"), true}, + {ce("gzip"), false}, + {ce("xaws-chunked"), false}, + {ce(""), false}, + {http.Header{}, false}, + } + for i, c := range cases { + if got := shouldDecodeAwsChunked(c.header); got != c.want { + t.Fatalf("case %d (%q) -> %v, want %v", i, c.header.Values("Content-Encoding"), got, c.want) + } + } + + stream := func(v string) http.Header { + h := http.Header{} + if v != "" { + h.Set("X-Amz-Content-Sha256", v) + } + return h + } + if !shouldDecodeAwsChunked(stream("STREAMING-AWS4-HMAC-SHA256-PAYLOAD")) { + t.Fatal("STREAMING- sha256 without CE must trigger decode") + } + if !shouldDecodeAwsChunked(stream("streaming-unsigned-payload-trailer")) { + t.Fatal("lowercase streaming- token must trigger decode") + } + if shouldDecodeAwsChunked(stream("UNSIGNED-PAYLOAD")) { + t.Fatal("UNSIGNED-PAYLOAD must not trigger decode") + } + if shouldDecodeAwsChunked(stream("")) { + t.Fatal("absent sha256 must not trigger decode") + } +} diff --git a/internal/engine/engine.go b/internal/engine/engine.go index 5ae49da..6b24ef3 100644 --- a/internal/engine/engine.go +++ b/internal/engine/engine.go @@ -1,6 +1,7 @@ package engine import ( + "bytes" "context" "crypto/sha256" "encoding/binary" @@ -500,20 +501,54 @@ func (e *Engine) serviceHandler(name string, svc manifest.Service) http.Handler var body []byte if r.Body != nil { + // S3 aws-chunked (SigV4 streaming) framing arrives as the + // request body with Content-Encoding: aws-chunked (or a + // STREAMING-* content hash without Content-Encoding from a + // malformed client). Peek the headers first: framed reads + // need headroom above bodyLimit for extensions/trailers, and + // the trigger must see every Content-Encoding value + // (headerMap keeps first-only, so Values is used here). + readLimit := bodyLimit + chunked := shouldDecodeAwsChunked(r.Header) + if chunked { + readLimit = bodyLimit + awsChunkedReadOverhead + } var err error - body, err = io.ReadAll(http.MaxBytesReader(w, r.Body, bodyLimit)) + body, err = io.ReadAll(http.MaxBytesReader(w, r.Body, readLimit)) if err != nil { // Never hand truncated data to a handler. Overflow is 413; // any other read failure is a 400. var maxErr *http.MaxBytesError if errors.As(err, &maxErr) { writeStatus(w, http.StatusRequestEntityTooLarge, - fmt.Sprintf(`{"error":"request body exceeds %d bytes"}`, bodyLimit)) + fmt.Sprintf(`{"error":"request body exceeds %d bytes"}`, readLimit)) return } writeStatus(w, http.StatusBadRequest, `{"error":"failed to read request body"}`) return } + if chunked && len(body) > 0 { + decoded, derr := decodeAwsChunked(body, bodyLimit, r.Header.Values("X-Amz-Decoded-Content-Length")) + if derr != nil { + var cerr *awsChunkedError + if errors.As(derr, &cerr) && cerr.status == http.StatusRequestEntityTooLarge { + writeStatus(w, http.StatusRequestEntityTooLarge, + fmt.Sprintf(`{"error":"request body exceeds %d bytes"}`, bodyLimit)) + } else { + writeIncompleteBody(w) + } + return + } + // Decoded bytes replace the body for profiles, rules, + // and handlers (r.Body + ContentLength included); the + // Content-Encoding and x-amz-content-sha256 headers are + // preserved untouched for SigV4. Double aws-chunked + // tokens decode once; aws-chunked over gzip stores the + // decoded gzip bytes as-is (no further decoding). + body = decoded + r.Body = io.NopCloser(bytes.NewReader(decoded)) + r.ContentLength = int64(len(decoded)) + } } // --- Active profile override layer ---