Skip to content

Casework: migration_locks_hosted_work_before_counting_it_for_the_drop is intermittent under load #1641

Description

@jeremi

CASEWORK_TEST_DATABASE_URL=... cargo test --locked -p registry-casework --features postgres-test --test postgres_transactions
sometimes fails migration_locks_hosted_work_before_counting_it_for_the_drop
with:

the migration must take its own exclusive lock on hosted work before
counting it, not rely on the shared lock a plain read takes, or a
concurrent writer could insert a row between the count and the drop unseen

It passes in CI (#1632's "Casework PostgreSQL transactions" job) and on a
fresh database (41/41). It failed only on repeated local runs on a loaded
machine, where settlement tests such as
a_live_execution_lease_refuses_settlement_and_writes_nothing flaked with
it. The test waits 5 s (500 polls of 10 ms) for the migration's lock, so a
slow migration start reads as a missing lock.

Ask: wait for the migration backend's lock with a longer bounded deadline
(for example polling pg_locks), so the test fails only when the lock is
really absent.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:caseworkRegistry Casework runtime, tooling, and clientsbugSomething isn't workingcriticality:p3Priority/criticality P3.triage:needs-implementationNeeds implementation.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions