Raising the socket idle timeout to 255s and adding a keepalive (#629, #630) means a stream no longer goes idle, so nothing on the Apollo side bounds how long a run can take. The only timers in platform/src now are the 5s SIGKILL escalation and the socket idle timer the keepalive deliberately defeats.
A Python service that wedges — an HTTP call with no timeout, a retry loop that never gives up — holds a socket, an interval and an interpreter for as long as the caller stays attached, and the keepalive is what keeps the caller attached. N concurrent requests to a wedged service pin N interpreters with no ceiling.
Cancellation covers the case where the caller leaves: the abort fires, the child gets SIGTERM. The gap is the caller that waits.
In practice Lightning bounds this from its side with APOLLO_REQUEST_TIMEOUT_MS, so today it is not unbounded — but that is Lightning defending itself, not Apollo. Any other client, or a change to that setting, and the ceiling is gone.
What it probably wants is a run deadline in bridge.ts: an AbortSignal.timeout(n) composed with the caller's signal, firing the same path cancellation already uses, with its own error type so it reads differently from a client disconnect.
The open question is the number. It has to sit above the slowest legitimate run — p99 of workflow generation is around 50-57s — and below whatever the callers use, or Apollo never gets to be the one that explains what happened.
Raising the socket idle timeout to 255s and adding a keepalive (#629, #630) means a stream no longer goes idle, so nothing on the Apollo side bounds how long a run can take. The only timers in
platform/srcnow are the 5s SIGKILL escalation and the socket idle timer the keepalive deliberately defeats.A Python service that wedges — an HTTP call with no timeout, a retry loop that never gives up — holds a socket, an interval and an interpreter for as long as the caller stays attached, and the keepalive is what keeps the caller attached. N concurrent requests to a wedged service pin N interpreters with no ceiling.
Cancellation covers the case where the caller leaves: the abort fires, the child gets SIGTERM. The gap is the caller that waits.
In practice Lightning bounds this from its side with
APOLLO_REQUEST_TIMEOUT_MS, so today it is not unbounded — but that is Lightning defending itself, not Apollo. Any other client, or a change to that setting, and the ceiling is gone.What it probably wants is a run deadline in
bridge.ts: anAbortSignal.timeout(n)composed with the caller's signal, firing the same path cancellation already uses, with its own error type so it reads differently from a client disconnect.The open question is the number. It has to sit above the slowest legitimate run — p99 of workflow generation is around 50-57s — and below whatever the callers use, or Apollo never gets to be the one that explains what happened.