Skip to content

fix(db): index large-value workflow_id FKs so workflow deletes don't seq-scan - #6136

Merged
TheodoreSpeaks merged 1 commit into
stagingfrom
feat/soft-delete-run-budget
Jul 31, 2026
Merged

fix(db): index large-value workflow_id FKs so workflow deletes don't seq-scan#6136
TheodoreSpeaks merged 1 commit into
stagingfrom
feat/soft-delete-run-budget

Conversation

@TheodoreSpeaks

@TheodoreSpeaks TheodoreSpeaks commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • execution_large_values.workflow_id and execution_large_value_references.workflow_id are ON DELETE SET NULL FKs (since 0212) with no index leading on that column
  • Postgres runs the SET NULL action as an AFTER ROW trigger — UPDATE ... SET workflow_id = NULL WHERE workflow_id = $1 once per deleted parent row — so every DELETE FROM workflow seq-scans both tables per workflow
  • The cost is proportional to table size x workflows deleted per statement, so it was invisible while the retention job kept the archived-workflow backlog small. That job last dispatched in prod on 2026-05-22 (per trigger.dev run history); whenever it resumes it faces ~10 weeks of accumulated backlog, and those deletes would blow the statement timeout
  • 0278 adds an index on both columns, built CONCURRENTLY below an embedded COMMIT per the runner convention in packages/db/scripts/migrate.ts
  • Neither index serves an app query — both exist to make the referential trigger index-driven

Type of Change

  • Bug fix

Testing

Applied the migration against a local DB: both indexes build and report indisvalid = t, and re-running the file is a no-op (already exists, skipping) — required since everything after the embedded COMMIT runs in autocommit and replays on failure. EXPLAIN on the trigger's predicate now shows Index Scan using execution_large_values_workflow_id_idx (forced with enable_seqscan = off; the local table only has ~2.4k rows).

bun run check:migrations origin/staging, bun run lint, and the full audit suite (check:boundaries, check:api-validation:strict, check:utils, check:zustand-v5, check:react-query, check:client-boundary, check:bare-icons, check:icon-paths, check:realtime-prune, skills:check, agent-stream-docs:check) all pass. bun run type-check exits 0.

Note for deploy: CONCURRENTLY does two table passes and waits out in-flight transactions, so the build will run for a while on prod-sized tables — worth checking indisvalid after, since a failed concurrent build leaves an INVALID index that the next IF NOT EXISTS silently skips.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Jul 31, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview Jul 31, 2026 10:37pm

Request Review

@cursor

cursor Bot commented Jul 31, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Production migration on large execution tables with long CONCURRENTLY builds; failed builds can leave INVALID indexes that IF NOT EXISTS skips—operational risk, not application logic change.

Overview
Adds migration 0278 with btree indexes on workflow_id for execution_large_values and execution_large_value_references. Those FKs are ON DELETE SET NULL (since 0212) but had no leading index, so each DELETE FROM workflow forced Postgres’s referential trigger to sequentially scan both tables per deleted row—likely to hit statement timeout once the soft-delete retention job starts hard-deleting archived workflows.

Indexes are built CREATE INDEX CONCURRENTLY IF NOT EXISTS after an embedded COMMIT, with lock_timeout cleared for the build and restored to 5s, matching the runner convention in migrate.ts. They are not for app queries—only to make the SET NULL trigger index-driven. Replay after failure is safe because post-COMMIT steps are idempotent.

Reviewed by Cursor Bugbot for commit 8746c4a. Bugbot is set up for automated code reviews on this repo. Configure here.

@greptile-apps

greptile-apps Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds concurrent indexes for the nullable workflow foreign keys used by large execution-value tables.

  • Declares workflow_id indexes for execution_large_values and execution_large_value_references in the Drizzle schema.
  • Adds migration 0271 with concurrent, post-COMMIT index builds.
  • Updates the Drizzle snapshot and migration journal.

Confidence Score: 4/5

This needs the concurrent-index replay path fixed before merging because a failed build can leave the migration complete without a usable index.

A partial CREATE INDEX CONCURRENTLY can leave an INVALID relation, and the subsequent IF NOT EXISTS replay skips it while the runner only warns, preserving the workflow-delete sequential scans this change is intended to eliminate.

Files Needing Attention: packages/db/migrations/0271_watery_scarlet_spider.sql

Important Files Changed

Filename Overview
packages/db/migrations/0271_watery_scarlet_spider.sql Adds the intended concurrent indexes, but replay after a partial build can accept an INVALID index and leave the performance defect unresolved.
packages/db/schema.ts Declares both workflow_id indexes with names and columns matching the migration.
packages/db/migrations/meta/0271_snapshot.json Records both indexes consistently with the schema and follows the previous snapshot ancestry.
packages/db/migrations/meta/_journal.json Correctly appends migration 0271 after migration 0270.

Reviews (1): Last reviewed commit: "fix(db): index large-value workflow_id F..." | Re-trigger Greptile

Comment on lines +12 to +13
CREATE INDEX CONCURRENTLY IF NOT EXISTS "execution_large_value_references_workflow_id_idx" ON "execution_large_value_references" USING btree ("workflow_id");--> statement-breakpoint
CREATE INDEX CONCURRENTLY IF NOT EXISTS "execution_large_values_workflow_id_idx" ON "execution_large_values" USING btree ("workflow_id");--> statement-breakpoint

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Invalid indexes survive migration replay

When either concurrent build fails after creating its catalog entry, the replayed IF NOT EXISTS statement skips the resulting INVALID index, allowing the migration to complete while workflow deletions continue sequentially scanning the corresponding large-value table.

Knowledge Base Used: Database schema and migrations (packages/db)

@TheodoreSpeaks
TheodoreSpeaks force-pushed the feat/soft-delete-run-budget branch from 28fdf04 to 8746c4a Compare July 31, 2026 22:32
@TheodoreSpeaks
TheodoreSpeaks merged commit 4ef7bba into staging Jul 31, 2026
27 of 28 checks passed
@TheodoreSpeaks
TheodoreSpeaks deleted the feat/soft-delete-run-budget branch July 31, 2026 23:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant