Backend payload signing doubles per-request upload memory — switch to UNSIGNED-PAYLOAD

[ServerSideHannes]

Problem

Every framed upload (client PutObject/UploadPart and the streaming-copy pump) buffers one internal part of ciphertext, and the backend transport adds a second full copy of that part when the request body is payload-signed: botocore reads the body to compute the SHA-256 payload hash for SigV4.

This is already documented in our own governor math — streaming_upload_peak() in s3proxy/crypto.py explicitly accounts for it:

"At peak it stacks the accumulated ciphertext (~part), one plaintext frame being sealed, and (for signed uploads) aiobotocore's copy of the part body for the HTTP request."

Measured (tracemalloc, per the docstring): 512 MB client part → 25.6 MB internal parts → 56 MB peak; 4 GB PutObject → 204 MB internal parts → 417 MB peak. Roughly half of the multi-frame term is the transport copy (2*part + FRAME → would drop toward 1*part + FRAME).

Fleet impact: proxy pods peak at 640–1020 MiB working set against a 192 MB governed budget (Thanos, Jul 6–10). Pod limits are 1 Gi and cannot be lowered while per-request honest peaks are this large.

Fix

Disable payload signing on the backend S3 client (UNSIGNED-PAYLOAD, e.g. payload_signing_enabled=False in the client config). RGW accepts unsigned payloads. Client-facing auth is unaffected — this is only the proxy→backend hop.

Expected result: multi-frame upload peak drops by ~1× part size (roughly halves for large parts). One-line config change plus governor formula updates in crypto.py (streaming_upload_peak, measured constants in docstrings/tests).

Validation

• tests/integration memory suites (memory_usage, memory_copy) + OOM Proof CI job — re-measure and update the calibrated constants.
• Note from prior incident: local ~/.aws/config can flip boto3 payload signing and change the proxy PUT path — CI (clean env) is authoritative.
• Verify against production RGW before rollout (some S3 implementations reject unsigned payloads over HTTP).

Context

Prereq for reducing pod memory limits (currently 1 Gi requests=limits × up to 30 pods). Related: #128.

Integrity requirement (no-regression condition)

Payload signing is currently the only transport-integrity check on the proxy→backend hop (verified: all MD5s in the upload handlers hash plaintext for client-facing ETags; nothing checks what the backend received). Removing signing alone would let an in-transit corruption be stored — never silently served (AES-GCM fails loudly on GET), but discovered only at restore time.

Therefore this change MUST pair UNSIGNED-PAYLOAD with ciphertext ETag verification: compute an incremental MD5 of the ciphertext while streaming (no extra buffering — this is the difference from the signing copy) and compare it against the backend's returned ETag for each internal part / PutObject. Mismatch → fail the part so the client retries. This is equal-or-better integrity than payload signing at ~zero memory cost (small CPU cost for MD5, cheaper than SHA-256).

Caveat: ETag == body-MD5 does not hold for SSE-KMS/customer-key buckets on AWS; it does hold for our RGW backend without SSE — assert this in the integration environment.

Backup impact & rollout

Expected impact on backups: none. Same requests, same retry behavior, same on-disk/object format — only the transport hop loses its extra body copy and gains the ETag check. Nothing already uploaded is affected.

Rollout: deploy between backup runs (not mid-run); verify RGW ETag semantics in integration first; rollback is a plain image revert (no data-format or state changes). Ship this before #128 — it is the low-risk half.