Skip to content

fix(engine): aws-chunked uploads stored the SigV4 framing as object content - #108

Merged
deblasis merged 4 commits into
mainfrom
pr/102-chunked
Oct 1, 2026
Merged

deblasis merged 4 commits into
mainfrom
pr/102-chunked

Conversation

@deblasis

@deblasis deblasis commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Fixes #102.

A default PutObject from AWSSDK.S3 4.0.102.4 sends STREAMING-AWS4-HMAC-SHA256-PAYLOAD framing with Content-Encoding: aws-chunked. The engine stored that framing as object content, so the chunk-size lines, the ;chunk-signature= extensions, the terminal zero chunk, and any trailers all landed in the object. Five bytes in, 298 bytes out. UseChunkEncoding=false on an otherwise identical request round-tripped correctly, which points away from the adapter and toward an SDK version difference.

Found by @jeremydixon22 driving the aws-s3-style adapter with the AWS .NET SDK.

internal/engine/engine.go read the body with io.ReadAll(http.MaxBytesReader(...)). Go's HTTP server strips Transfer-Encoding: chunked, but Content-Encoding: aws-chunked is framing inside the payload itself, and no decoder existed in Go or Starlark. The request authenticated before it stored the framed bytes, because adapters/aws-s3-style/scripts/lib.star:457-467 already accepted the STREAMING-* content hash verbatim.

What changed

internal/engine/chunked_aws.go decodes the framing after the body read and before the profile layer, so rules, profiles, and handlers all observe decoded bytes. Content-Encoding and x-amz-content-sha256 pass through untouched, since SigV4 signs them; deleting either would break signature verification on any SDK that includes them in its signed set.

Decoding is triggered by an aws-chunked token in Content-Encoding (comma-split, case-folded, multi-value aware) or by a STREAMING-* content hash without that header. The read limit for the still-framed body is max_body_bytes plus 2 MiB of framing headroom, so chunk signatures and trailers do not cause a spurious 413.

Malformed, truncated, or over-limit framing returns 400 IncompleteBody without echoing request bytes. Limits: 4 KiB per chunk or trailer line, 10,000 data chunks, 32 trailers totalling 8 KiB, and decoded output bounded by max_body_bytes. x-amz-decoded-content-length is verified when exactly one valid value is present and ignored otherwise, never pre-allocated from. The parser accepts uppercase hex, treats bytes after the first ; as an opaque extension, and reads exactly size bytes then CRLF, so binary payloads containing CRLF are safe.

Deviations

Per-chunk signatures are not verified; the header signature is. Streaming checksum trailers are discarded. Both are recorded in conformance/matrix.yaml:157-158.

Verification

$ env -u GOROOT go test -race ./internal/engine -run 'TestAWSChunked|TestBodyLimit' -v
--- PASS: TestAWSChunkedDecode
--- PASS: TestAWSChunkedParser (15 subtests)
--- PASS: TestAWSChunkedTrigger

TestAWSChunkedParser covers single and multi-chunk bodies, binary payloads containing CRLF and NUL, uppercase hex, OWS around the chunk line, both STREAMING-* tokens, trailers, the count and size caps, decoded-over-max_body_bytes, and malformed input including bare LF, a +-prefixed size, and a trailer with no colon.

Round-trip through AWSSDK.S3 4.0.102.4, 5 bytes in and 5 bytes out:

{ "awsChunkSignaturesStoredAsContent": false, "rawHttpBodyLength": 5, "headContentLength": 5 }

stunt adapter lint adapters/aws-s3-style is clean and just conformance-matrix regenerates with no drift.

@deblasis
deblasis merged commit e9dfb60 into main Oct 1, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

aws-chunked uploads stored the SigV4 framing as object content

1 participant