fix(proxy): gate /readyz on a decodable Fetch round-trip

[kamir]

What

/readyz on the proxy answers two questions instead of one:

1. Is a backend known? (the existing shallow check, unchanged, now haveBackend)
2. Does that backend actually serve Fetch? (new, a decodable Fetch round-trip)

KAFSCALE_PROXY_READYZ_FETCH_PROBE=false falls back to the shallow check.

Why

The proxy is a full Fetch codec: it decodes, merges and re-encodes broker Fetch
responses. A proxy-to-broker Fetch serialization or connection-state mismatch
therefore breaks consume while Metadata and ListOffsets keep answering
normally.

In that state readiness based on question 1 alone reports ready, Kubernetes
keeps the pod in the Service, and every consumer silently receives zero
records. The failure is invisible to kubectl get pods and to any health check
that only asks whether a process is up.

The probe asks for one partition of a deliberately unknown topic ID, so it
needs no real topic and reads no data; a correct broker answers
UNKNOWN_TOPIC_ID. It uses a fresh connection on purpose, which also detects a
stale or half-open pooled connection that cached state cannot.

The part worth reviewing: framing

forwardToBackend prepends the frame length itself, and encodeFetchRequest
already strips the size prefix that kmsg's AppendRequest emits. The probe
payload must therefore be the bare request header plus body.

Stripping a second time removes the API key and version, four bytes. The
request is then not a malformed Fetch, it is a different API: the broker reads
the correlation ID as the API key. A broker that cannot parse a request header
closes the connection, so the probe reports EOF and the failure reads like a
broken broker rather than a broken probe.

buildFetchProbePayload is split out so the encoding is unit-testable without a
backend.

Startup ordering

With the probe enabled the proxy stays NotReady until a broker serves Fetch. An
installer that waits for proxy readiness before creating the cluster resource
that produces the broker will deadlock. Create the broker first, or do not gate
the install on proxy readiness. Documented in docs/operations.md.

Tests

Test	States
TestBuildFetchProbePayloadIsSingleFramed	the payload parses back as Fetch v13 with the probe's topic ID, one partition, ReplicaID -1
TestBuildFetchProbePayloadRejectsDoubleStrip	a payload that has lost its four header bytes must NOT parse as Fetch, so the test above cannot silently stop distinguishing the bug
TestEnvBoolDefault	parsing, including that an unparseable value falls back rather than flipping a gate

Both framing tests were checked against a deliberately broken encoder:

with the double strip:
  --- FAIL: TestBuildFetchProbePayloadIsSingleFramed
      probe payload is not parseable as a request:
      read client id: insufficient bytes: need 26227 have 98
without it:
  ok  github.com/KafScale/platform/cmd/proxy

need 26227 have 98 is what the broker sees before it closes the connection.

Verification

• gofmt -l . clean, go vet ./cmd/... ./pkg/... clean
• go test ./cmd/proxy ok
• go test ./... 22 packages ok, 0 failures

Verified end to end on a single-node k3s edge deployment: with the probe
correctly framed the proxy reaches Ready and helm --wait completes; produce
and consume of 500 messages through the proxy round-trip with no gaps,
duplicates or checksum errors.

Scope

Three files, one concern. No behaviour change when
KAFSCALE_PROXY_READYZ_FETCH_PROBE=false.

[PaxMachinaOne]

🤖 PaxMachina automated review (worker-code)

The Fetch probe’s timeout does not bound backend I/O.

• Blocking: cmd/proxy/main.go:481 — probeCtx is not applied as a connection deadline, while forwardToBackend performs context-unaware blocking reads/writes. A backend that accepts TCP but never returns a complete frame can therefore hang /readyz indefinitely, accumulating connections and handler goroutines as probes repeat. Set the connection deadline from probeCtx or make the forwarding I/O cancellation-aware.