Repository navigation
fix: mark runs failed when the SQS send fails instead of leaving them queued - #19
Merged
Merged
Conversation
… queued
approve-rules and auto-validate move a run to 'queued' and then send to the
executor queue; profile-complete moves it to 'awaiting_ai' and sends to the
AI jobs queue. When SendMessage failed, approve-rules and auto-validate only
logged and returned 200, and profile-complete threw a bare 500. Either way
the run sat in its in-between status with no message until reconcile-runs
failed it 20 minutes later.
Now each route marks the run failed with a message the run page shows and
returns 503 { error, run_status: 'failed' }. The shared helper
(src/lib/queue-failure.ts) updates only if the run is still in the status
the route set, because SendMessage can fail after SQS accepted the message
and a worker may already have claimed the run. suggest-transforms no longer
overwrites that message with 'auto-validate error 503: ...'.
Tests: unit tests for all three routes, and an integration test that sends
to a LocalStack queue that does not exist.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
When SendMessage fails after a route has moved a run forward, the route now marks the run
failedwith a clearerror_messageand returns503 { error, run_status: "failed" }. It no longer leaves the run stuck.POST /api/approve-rulesSQS_QUEUE_URL)queuedqueuedPOST /api/auto-validate/[runId]queuedqueuedPOST /api/webhooks/profile-complete(whenAI_QUEUE_ENABLED=true)awaiting_aiawaiting_aiThe shared helper is
src/lib/queue-failure.ts. The review page already shows the response'serror, and the run page showserror_messageon failed runs.Look hardest at
WHERE id = $1 AND status = <status the route set>). SendMessage can fail even though SQS accepted the message, for example on a timeout. If the executor has already claimed the run, it is left alone. If the run was marked failed first, the executor's own conditional claim skips it. The helper never throws: if the DB update fails, the route still returns 503, and reconcile-runs remains the backstop.error_messagewithauto-validate error 503: …. That UPDATE now skips runs that are alreadyfailed, so the clearer message survives.Tests
approve-rules/route.test.ts(extended, including a DB failure while marking the run failed), plus newauto-validate/[runId]/route.test.tsandwebhooks/profile-complete/route.test.ts.approve-rules-claim.test.tssends to a LocalStack queue that doesn't exist, so the SendMessage error is real, and asserts the run isfailedwith the message.After merge
This is an app-only change, so Vercel's deploy on merge is enough. Docs: the runbook row and the README limitation are updated.