How this surfaced
Reviewing the outbox relay, I pushed on why it polls:
Polling seems heavy given it may not have records for hours to look at.
The answer I initially reached for was that polling is cheap and the service is never idle for long, because roadside toll traffic is continuous.
That premise is wrong for this deployment. A large share of transactions reach us as batch files — reciprocity files from interoperability peers, and image-review output. The contract names a file-based batch loader as a first-class producer.
So the real workload is not a steady trickle. It is:
idle ── idle ── idle ── idle ── ▓▓▓▓▓ FLOOD ▓▓▓▓▓ ── idle ── idle
Every sizing decision in the relay was made against the wrong shape, and this ticket records what changes as a result.
What the relay does today
A fixed 2-second poll loop. Each iteration issues three queries plus a transaction:
PendingCount — for the backlog gaugeOldestPending — for the lag gaugeBEGIN → claim (FOR UPDATE SKIP LOCKED) → ROLLBACK when empty
Two replicas × 3 queries every 2s ≈ 130,000 queries/day, the overwhelming majority finding nothing.
Two tasks also run 24/7 on Fargate to do work that arrives in bursts.
What actually changes
1. 🔴 Serial publishing is the real bottleneck
This matters far more than the poll interval, and it was invisible while I assumed a steady trickle.
The relay claims up to 100 events, then publishes them one at a time in a loop. For a 360,000-transaction file that is 360,000 sequential round trips to SQS.
SendMessageBatch sends 10 per call — roughly a 10× improvement in drain rate, and the burst is the workload.
Note the poll interval is irrelevant here: after a non-empty pass the relay immediately polls again, so the 2 seconds never applies once a burst has landed. Throughput is the constraint, not cadence.
2. Adaptive backoff
Long idle is the normal state, not an incident. A fixed 2s interval is waste.
100ms while draining, backing off to ~30s when idle. Cheap to implement, and it improves burst latency while removing most of the idle queries.
3. Fold the gauge queries into the claim
Two of the three queries per tick exist only to feed metrics gauges. They can be derived from the claim itself, or sampled on a much slower timer than the drain loop.
💰 The question worth asking: should this run at all when idle?
Two Fargate tasks running continuously for a workload that arrives in bursts is paying 24/7 for something used a few times a day. The spend is modest — order of $25/month at current sizing — but the principle is the point, and it scales badly across a fleet of services shaped like this one.
The obvious suggestion is "trigger the relay from SQS instead of leaving it running." Worth stating carefully, because there is a trap in it.
Why "just event-trigger it" is not a complete answer
Nothing can natively trigger off an outbox insert. The outbox is a Postgres table. There is no hook from a row insert to Lambda or ECS.
Any design where the ingest API tells the relay "there's work" reintroduces exactly the dual-write problem the outbox exists to eliminate: commit succeeds, signal is lost, nobody drains, and no error appears anywhere. If the signal is the only trigger, a lost signal is a lost transaction.
So a poll must remain as the correctness backstop. The question is not poll or event — it is how slow can the poll be once something else handles the common case.
That reframing is the important part of this ticket.
Options
|
Approach |
Idle cost |
Burst latency |
Correctness risk |
Complexity |
| A |
Always-on + adaptive backoff |
Unchanged (~$25/mo) |
Low |
None |
Trivial |
| B |
Always-on + LISTEN/NOTIFY, slow poll backstop |
Slightly lower |
Lowest |
None — poll backs it up |
Low |
| C |
Scale-to-zero, woken by an SQS message when a file lands |
Near zero |
Cold-start delay |
Needs a periodic backstop run |
Medium |
| D |
CDC (logical replication / Debezium) |
Near zero |
Lowest |
New infrastructure to operate |
High |
Recommendation: A now, C if the economics justify it.
Adaptive backoff is a few lines and removes most of the waste. Option C genuinely eliminates idle spend and fits the batch shape well — file lands, a notification wakes the relay, it drains, it stops — but it needs a scheduled backstop run so a missed wake-up degrades to late rather than lost.
Option D is the textbook event-driven outbox and is the right answer at much larger scale. It is not worth the operational surface here.
Open questions for the team
- How large are the batch files, and how often do they land? This decides whether the drain window is minutes or hours, and whether option C's cold start matters at all.
- Is there a signal when a file arrives that the relay could subscribe to — or does the loader simply start pushing?
- Does the resolution pipeline have a latency expectation? If it is happy with "within the hour," option C is comfortable. If anything expects seconds, the always-on relay stays.
Acceptance criteria
Related
How this surfaced
Reviewing the outbox relay, I pushed on why it polls:
The answer I initially reached for was that polling is cheap and the service is never idle for long, because roadside toll traffic is continuous.
That premise is wrong for this deployment. A large share of transactions reach us as batch files — reciprocity files from interoperability peers, and image-review output. The contract names a file-based batch loader as a first-class producer.
So the real workload is not a steady trickle. It is:
Every sizing decision in the relay was made against the wrong shape, and this ticket records what changes as a result.
What the relay does today
A fixed 2-second poll loop. Each iteration issues three queries plus a transaction:
PendingCount— for the backlog gaugeOldestPending— for the lag gaugeBEGIN→ claim (FOR UPDATE SKIP LOCKED) →ROLLBACKwhen emptyTwo replicas × 3 queries every 2s ≈ 130,000 queries/day, the overwhelming majority finding nothing.
Two tasks also run 24/7 on Fargate to do work that arrives in bursts.
What actually changes
1. 🔴 Serial publishing is the real bottleneck
This matters far more than the poll interval, and it was invisible while I assumed a steady trickle.
The relay claims up to 100 events, then publishes them one at a time in a loop. For a 360,000-transaction file that is 360,000 sequential round trips to SQS.
SendMessageBatchsends 10 per call — roughly a 10× improvement in drain rate, and the burst is the workload.Note the poll interval is irrelevant here: after a non-empty pass the relay immediately polls again, so the 2 seconds never applies once a burst has landed. Throughput is the constraint, not cadence.
2. Adaptive backoff
Long idle is the normal state, not an incident. A fixed 2s interval is waste.
100ms while draining, backing off to ~30s when idle. Cheap to implement, and it improves burst latency while removing most of the idle queries.
3. Fold the gauge queries into the claim
Two of the three queries per tick exist only to feed metrics gauges. They can be derived from the claim itself, or sampled on a much slower timer than the drain loop.
💰 The question worth asking: should this run at all when idle?
Two Fargate tasks running continuously for a workload that arrives in bursts is paying 24/7 for something used a few times a day. The spend is modest — order of $25/month at current sizing — but the principle is the point, and it scales badly across a fleet of services shaped like this one.
The obvious suggestion is "trigger the relay from SQS instead of leaving it running." Worth stating carefully, because there is a trap in it.
Why "just event-trigger it" is not a complete answer
Nothing can natively trigger off an outbox insert. The outbox is a Postgres table. There is no hook from a row insert to Lambda or ECS.
Any design where the ingest API tells the relay "there's work" reintroduces exactly the dual-write problem the outbox exists to eliminate: commit succeeds, signal is lost, nobody drains, and no error appears anywhere. If the signal is the only trigger, a lost signal is a lost transaction.
So a poll must remain as the correctness backstop. The question is not poll or event — it is how slow can the poll be once something else handles the common case.
That reframing is the important part of this ticket.
Options
LISTEN/NOTIFY, slow poll backstopRecommendation: A now, C if the economics justify it.
Adaptive backoff is a few lines and removes most of the waste. Option C genuinely eliminates idle spend and fits the batch shape well — file lands, a notification wakes the relay, it drains, it stops — but it needs a scheduled backstop run so a missed wake-up degrades to late rather than lost.
Option D is the textbook event-driven outbox and is the right answer at much larger scale. It is not worth the operational surface here.
Open questions for the team
Acceptance criteria
Related