snapshot s3: parallel downloads with a bytes-in-flight budget and --download-* flags

[mbevc1]

Context

Following #1155 next step is to change kosli snapshot s3 behaviour and stop writing bucket objects under their keys on the operator's filesystem. Each object will be downloaded to an anonymous temp file, hashed and removed, and the fingerprint is computed from the object keys and content digests as a virtual directory tree, reproducing what digest.DirSha256 gives the same tree on disk. That change will replace the key containment added in #1155 and carry its own ADR. It'll keep downloads sequential, one object at a time, exactly as today.

Following this we should also add bounded parallel downloads and the flags to tune them. It is a performance change only: fingerprints, the key rule and the .kosli_ignore handling are unchanged.

Design

• Fixed worker pool. Exactly concurrency workers pull object indexes from a channel fed in listing order. Goroutines are bounded by the concurrency, never by the bucket size; a goroutine-per-object fan-out parks one stack per object and is not acceptable.
• Bytes-in-flight budget. A weighted semaphore caps the sum of the listed sizes of the objects downloading at once, which caps temp disk. An object larger than the whole budget takes all of it and runs alone; it is never refused.
• Cancellation. The first transport error cancels a shared context: in-flight multipart transfers stop, the producer stops feeding, the workers drain, and the snapshot fails with the key named. Never a partial fingerprint.
• Ordering. Results are written by listing index, so the manifest is deterministic however downloads interleave. The root .kosli_ignore is still fetched first, since its rules decide what else to download.
• Two flags on kosli snapshot s3, also settable as KOSLI_DOWNLOAD_CONCURRENCY and KOSLI_DOWNLOAD_BUDGET:
  • --download-concurrency (int, default 8): objects downloading at once.
  • --download-budget (size, default 512M): a bare number is megabytes; K, M, G or T with an optional B, case-insensitive, picks the unit, so 512, 512M, 512MB and 0.5G are the same. Units are binary, matching how Lambda's /tmp is expressed. Validated before any request.
• Transfer manager part concurrency lowered from the SDK default of 5 to 3, so object-level times part-level stays modest (8 × 3 = 24 connections at most).

Behaviour changes to call out in review and release notes

• Peak temp disk rises from one object to the budget, by design. TMPDIR chooses the filesystem.
• Connections rise from about 5 to about 24 in the worst case. Throttling is handled by the existing adaptive retryer, whose token bucket is shared across all goroutines.

Slices

Each is independently mergeable and leaves the command working.

• Parallel downloads with a count bound. Worker pool, results by index, first error cancels the rest. Tests: one call per object, concurrency bound respected and actually exercised, fingerprint independent of completion order (against digest.DirSha256 of the same tree), a transport error stops remaining work, goroutines during a 2000-object run stay within concurrency plus a small constant. Run under -race.
• Bytes-in-flight budget. Weighted semaphore on listed sizes; oversized object runs alone. Tests: budget binds before the count bound, oversized object is the only download in flight, run completes.
• Transfer manager part concurrency to 3.
• --download-concurrency and --download-budget flags. Size parser with a table test covering bare numbers, every unit spelling, decimals, and rejections (empty, zero, negative, unknown unit, too large). Validation in PreRunE with messages naming the flag. Flag default string pinned by a test to the aws default. Both flags registered in cmd/kosli/testdata/empty-flag-audit-coverage.json. cmdTestCase rows for the valid and invalid forms.
• Verification that needs infrastructure. The cmd/kosli S3 command suite against the local Kosli server, and the credential-gated TestGetS3Data case against kosli-cli-public.
• Release note covering the new flags, the higher default disk and connection use, and how to lower both.

Related

• fix(snapshot s3): reject object keys with ".." segments and never overwrite a downloaded object #1155 — the interim key-containment fix that the pending change replaces.
• feat(snapshot s3): fingerprint buckets from metadata #1069 — --fingerprint-source metadata; unaffected, since metadata mode does not download.

[mbevc1]

Two scale notes from the review of #1178, for sizing this work:

• The whole object listing is materialised before the first download, because collision detection and .kosli_ignore resolution need the complete key set. Per surviving object the run holds the listing entry, the key-to-path map, the per-path key index, the manifest and one virtual tree node, roughly a few hundred bytes each. The byte budget should be sized with that fixed cost in mind.
• The virtual tree is built and the ignore rules are resolved twice per snapshot: once to decide which objects need downloading and once to compute the digest. A single prepared-tree handle that answers both questions would remove the duplication, and belongs with the parallel download slices rather than in feat: fingerprint S3 buckets from a virtual tree #1178.

---

Generated by Claude Code