Repository navigation
Wait for Aurora to resume from auto-pause instead of failing - #22
Merged
Merged
Conversation
Both EventBridge rules now run every 4 hours (cron(0 */4 * * ? *)) so the Serverless v2 cluster can auto-pause; it never paused under the 5-minute reconciler (about $45 a month). The first request after a pause has to wait for the resume (~15s, 30s+ after a day). - src/lib/db-resume.ts: retry Data API calls only on DatabaseResumingException (the request was cancelled before it ran, so writes are safe to retry), with 1/2/4/8s backoff for up to 35s and one warn log line. Wraps ExecuteStatement, Begin/Commit/RollbackTransaction in src/lib/db.ts. The pg test driver is unchanged. Other errors are never retried. - profiler, executor, drift Lambdas (psycopg2, not the Data API): retry a failed connect for up to 35s, except authentication failures. - reconcile-runs maxDuration 30 -> 60, purge-guests 60 -> 120. - Docs: new schedule, why, and the resume behaviour (operations.md).
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both EventBridge rules (
cleanstack-reconciler-5min,cleanstack-purge-guests-hourly) now run oncron(0 */4 * * ? *). The clusterdatabase-1(Serverless v2, min 0 ACU, auto-pause after 300 s) never paused under the 5-minute reconciler, which cost about $45 a month. Now it pauses, and the first request after a pause fails while the cluster resumes (~15 s, 30 s+ after a pause longer than 24 h). This PR makes callers wait instead.Web app (
src/lib/db.ts, RDS Data API)src/lib/db-resume.tswithResumeRetry(). It wrapsExecuteStatement,BeginTransaction,CommitTransactionandRollbackTransaction.DatabaseResumingException. The Data API reference describes this error as "a request was cancelled because the Aurora Serverless v2 DB instance was paused… wait a few seconds and try again", so the statement never ran and retrying writes is safe. It is a 400 client fault, so the SDK's own retry strategy doesn't retry it.BadRequestException("Communications link failure") is documented for Serverless v1 only and is not retried. Nor areStatementTimeoutException,DatabaseUnavailableExceptionor SQL errors, which can happen after a statement was sent.database is resuming from auto-pause; retrying Data API callwithop.DB_DRIVER=pg) is unchanged.maxDurationgoes from 30 to 60 for/api/cron/reconcile-runsand from 60 to 120 for/api/cron/purge-guests. Both now almost always hit a paused cluster. The other routes with an explicit value already have 60–300 s.Lambdas
None of the Lambdas use the Data API. profiler, executor and drift connect with psycopg2 and IAM auth, and ai-trigger only calls the app over HTTP. A direct connection to a paused cluster normally waits for the resume. AWS still recommends retrying failed connects, because a burst of connections during a resume can fail. Each of those three handlers gets the same small
_connect_with_resume_retry(), used byget_db_conn():psycopg2.OperationalErrorraised byconnect, where no SQL has run. Authentication failures are not retried.[db] connect failed, database may be resuming from auto-pause.Tests
src/lib/db-resume.test.tscovers retry then succeed, no retry for other errors, giving up after the budget, and the single log line.src/lib/db.test.tschecks that every Data API command goes through the retry.lambdas/tests/test_db_resume_retry.pyruns the same cases against each of the three handlers.Docs
docs/operations.mdhas a new "Aurora auto-pause" section and updated runbook rows: a stuck run can now take about 4 h 20 min to be failed, and the purge erases about 120 guests a day.docs/deployment.md, the README anddocs/security-model.mdhave the new schedule and why.After merge: redeploy the profiler, executor and drift Lambdas with the zip-swap. ai-trigger is unchanged.