Skip to content

refactor(eggs): put egg operations behind an IEggOperationsModule contract - #1028

Merged
mforce merged 8 commits into
mainfrom
refactor/853-egg-operations-contract
Oct 3, 2026
Merged

mforce merged 8 commits into
mainfrom
refactor/853-egg-operations-contract

Conversation

@mforce

@mforce mforce commented Oct 2, 2026

Copy link
Copy Markdown
Owner

Closes #853

Egg Operations now sits behind a contract: daily entries, grades, lots and the egg ledger. It is the fifth module after Finance (#849), Farm (#851), Flock Management (#852) and General Inventory (#855). No route, response, status, query, audit row or schema changes.

A real coverage gap on main: nothing pinned the egg lot lock order

Every FOR UPDATE read over EggLots orders by (ProductionDate, Id), so a confirm, a sale void and a daily-entry void that touch the same lots queue instead of deadlocking. No test noticed when that order changed. On main at 54a5f298, with both GetByIdsLockedAsync and GetByDailyEntryLockedAsync reversed to ORDER BY "ProductionDate" DESC, "Id" DESC, all 1881 integration tests passed, and so did the 42 tenant-bypass tests (the allow-list keys raw SQL by signature, not by SQL text).

The first commit adds EggLotLockOrderTests, before any structure moves. Each test holds the canonically later lot in a second connection and waits until the request is parked on it (pg_blocking_pids). It then tries the earlier lot with FOR UPDATE NOWAIT, and that try must fail, because the request already holds that lot. Each reversed order turns its test red, with the message the request must already hold the earlier lot while it waits on the later one:

Read reversed Path Test that turns red
GetByIdsLockedAsync sale void VoidSale_HoldsTheOlderSourceLotWhileWaitingOnTheNewer
GetByDailyEntryLockedAsync daily-entry adjust/void VoidDailyEntry_HoldsTheEarlierLotWhileWaitingOnTheLater
GetAvailableFifoLockedAsync sale confirm ConfirmSale_HoldsTheOlderLotWhileWaitingOnTheNewer (and VoidSale_…, because its setup's confirm then draws from one lot only)

Each mutation was reverted and the binaries rebuilt. The pin passed 5 of 5 runs on the unchanged code.

Checkpoint decisions (owner-approved)

Rationale: docs/decisions/853-egg-operations-contract.md. AGENTS.md's #849 paragraph now names Egg Operations and the lock-order rule.

Reach

Before (main 54a5f29, placeholder contract) After
Adapter crossings into Egg Operations through a non-contract type 37 (21 types, 24 adapter rows) 0
Adapter rows reaching Egg Operations (matrix Platform -> EggOperations) A (24) A (25), adding DailyEntryLockSweep.LockDueEntriesAsync
Adapters walked / non-empty rows 403 / 150 403 / 151
#850 reads of Egg Operations tables from outside it, unregistered 34 0
...admitted by the 4 listed repository implementations 29
...moved into the module (the sweep's read) 1
...registered compatibilityExceptions rows (Platform, deleteWhen #858) 5 (4 existing seeder reads + the natural-key check)

No edges were added or removed. The adapter pruning list (Loosenable) is empty. The regenerated coupling-matrix.md changes only the A (24) → A (25) cell.

Change map

BEFORE                                         AFTER
DailyEntryEndpoints (7 adapters)               DailyEntryEndpoints / EggGradeEndpoints / StockEndpoints
EggGradeEndpoints (7 adapters)                  └─► IEggOperationsModule ◄── new front door
StockEndpoints (4 adapters)                          └─► EggOperationsModule (only forwards)
 ├─► 8 handlers                                           ├─► 8 handlers (bodies unchanged)
 ├─► 4 repositories                                       ├─► LockDueDailyEntriesHandler (sweep loop, moved)
 └─► DailyEntry / EggGrade (ToResponse)                   ├─► 4 repositories
                                                          └─► returns 4 *Details records + StockByGrade
SaleEndpoints (2 adapters) ─► IEggGradeRepository    SaleEndpoints ─► IEggGradeLookup ◄── new
DailyEntryLockSweep ─► db.DailyEntries (read+lock)  DailyEntryLockSweep ─► IEggOperationsModule
                                                     (its db.Accounts read stays a #858 row)
DemoDataSeeder / SimulationDataSeeder          DemoDataSeeder / SimulationDataSeeder
 ├─► Record/SubmitDailyEntryHandler             ├─► IEggOperationsModule
 ├─► IEggGradeRepository, IDailyEntryRepository └─► db.* Egg Operations reads (5 rows, deleteWhen #858)
 └─► db.* Egg Operations reads
RecordFeedUsageHandler / RecordWaterUsageHandler     RecordFeed/WaterUsageHandler
 └─► IDailyEntryRepository                            └─► IDailyEntryLookup ◄── new
Commerce: Confirm/VoidSale, Create/UpdateProduct, AddOrderItem
 └─► IEggLotRepository, IEggInventoryMovementRepository, IEggGradeRepository, EggLot   (UNCHANGED, #854)

module-ledger.json
  owners.EggOperations.contract        = 3 interfaces, 8 records, 6 commands + GradeQuantityDto,
                                         4 responses, 4 Domain.Eggs enums (26 types)
  owners.EggOperations.implementations = DailyEntryRepository, EggGradeRepository,
                                         EggInventoryMovementRepository, EggLotRepository
  compatibilityExceptions              + 5 seeder rows (Platform, #858)
  adapters                             + DailyEntryLockSweep.LockDueEntriesAsync -> EggOperations
  edges                                unchanged; GeneralInventory -> EggOperations reason names IDailyEntryLookup
  foreignKeys                          unchanged; 3 reasons corrected (BirdMovements, FeedUsages, WaterUsages)
coupling-matrix.md                     regenerated: Platform -> EggOperations A (24) -> A (25)

Files this PR shares with open slices end in this shape:
  SaleEndpoints (#854): only the two IEggGradeRepository parameters became IEggGradeLookup.
  DemoDataSeeder, SimulationDataSeeder (#858): constructors take IEggOperationsModule.
  DailyEntryLockSweep (#858): per-account body calls the contract; RunAsync's Accounts read unchanged.
  module-ledger.json, coupling-matrix.md, AGENTS.md, DI registrations.

Verification

Test counts were measured here, not copied from elsewhere. All suites pass, and no existing test file was edited: the only test-file changes are two new files and the ledger data.

Project main 54a5f29 This branch 3f2584f
Domain 495 495
Application 661 669 (+8 EggOperationsModuleTests)
AppHost 10 10
Integration 1881 1884 (+3 EggLotLockOrderTests)

dotnet build Cluckwork.sln is clean with warnings as errors. The branch base is 54a5f298. main has since gained #1025, which adds two test files and overlaps nothing here; CI runs on the merge ref.

EF model digest. The SHA-256 of the design-time Model.ToDebugString(LongDefault) is 45F550E426163CEBEFF36E509CFC6E53B36D26D3D89C72E0F728855BCC8FB9AE, on main 54a5f29 and on this head. The two 1486-line dumps are byte-identical.

Invariants. Counts below are from the full integration run at 3f2584fe; every test passed and none was edited.

Invariant Suites (passed)
Parallel submit, adjust and void races and their 409s SubmitDailyEntryTests (13, incl. ParallelSubmits_ExactlyOneWins_NoDuplicateLots), DailyEntryAdjustTests (13, incl. ParallelAdjusts_SameBaseVersion_ExactlyOneWins, ParallelAdjustAndVoid_SameBaseVersion_ExactlyOneWins), EggLotWriteOffTests (17, incl. TwoRacingWriteOffs_OneWins_OneRefused_LedgerBalances), EggGradeManagementTests (6), EggGradeFloorRaceTests (1)
Canonical lock order (ProductionDate, Id) EggLotLockOrderTests (3, new), EggLotConcurrencyTests (2), VoidSaleTests (7, incl. VoidRacingConfirm_OnSameLots_NeverLosesOrDuplicatesStock)
Cached balance equals the ledger sum EggLedgerTests (6, incl. Ledger_RebuildsTheCachedBalance_AcrossTheFullLifecycle), EggLotWriteOffTests
Sold-quantity floors DailyEntryAdjustTests (Adjust_ShrinkBelowSold_Is422_AndNothingChanges, Void_WithSoldStock_Is422), EggLotWriteOffTests (reconcile limits)
Lot eligibility (#396 condition lots, withdrawal restriction) QualityEggTests (12), WithdrawalRestrictedLotTests (1)
Rollback after a partial write MortalityLedgerAtomicityTests (2), AuditTests (8, incl. MidTransactionFailure_WritesNothing_EventualVoidWritesOne), EggLedgerTests.FailedOperations_WriteNoMovements, SubmitDailyEntryTests.Submit_PartialGrades_Fails_NoLotsCreated
Audit rows unchanged (Record, Submit, Adjust, Void, grade writes, lot movement) AuditProvenanceTests (48, incl. DailyEntrySubmit_WritesAnAuditEvent), AuditTests (8), EggGradeManagementTests.Grade_Create_WritesAuditEvent, EggLotWriteOffTests.WriteOff_WritesAuditRow
Tenant and flock scope, HTTP contract FlockScopeTests (16), TenantIsolationTests (2), TwoFarmIsolationMatrixTests (1), ReadEndpointTests (13), RoleMatrixTests (18), AdminGatingTests (7), EggGradeLowStockFloorTests (9), SalesProductTests (25)
Lock sweep DailyEntryAdjustTests.LockSweep_* (2), SimulationDeepSeedDrainTests (2)
Feed and water provenance, feed lock order FeedUsageTests (13), WaterUsageTests (7), FeedUsageLockOrderTests (2)
Seeders DemoSeedTests (2), SimulationSeederTests (39)
EF model digest identical, above

Mutations. Each was run against a clean build and reverted afterwards, and every one turned red:

Mutation Red test
Lock order reversed in each of the three reads see the first table
FarmId ↔ HouseId in the entry copy EggOperationsModuleTests.GetDailyEntry_…, ListDailyEntries_…
CrackedGradeId ↔ DirtyGradeId in the entry copy same two
QuantityProduced ↔ QuantityAvailable in the lot copy ListLots_ForwardsTheFiltersAndCopiesEveryField
Lot existence check dropped before the movement read ListLotMovements_CopiesEveryField_UnknownLotIsNullWithoutAMovementRead
Active-grade list drops its farm Grades_CopyEveryField_AndForwardTheFarm
IDailyEntryLookup swaps farm and house DailyEntryLookup_ForwardsTheNaturalKeyAndReturnsTheId
IDailyEntryLookup always returns null FeedUsageTests.Usage_WithExistingDailyEntry_StampsTheLink, WaterUsageTests.Record_WithExistingDailyEntry_StampsTheLink_AndUpdateKeepsIt
IEggGradeLookup returns no names SalesProductTests.OrderItems_ReadCurrentGradeName_EvenWhenInactive (both cases)
LockDueDailyEntriesHandler without its save DailyEntryAdjustTests.LockSweep_LocksOldSubmittedEntries_…, LockSweep_StillLocksEntries_ForASuspendedFarm
Seeder natural-key check misses (wrong date) SimulationSeederTests.SimulationSeed_ProductionHistory_IsIdempotent_NoDuplicateEntriesAfterTwoHostStarts
IEggLotRepository eggLots added back to StockEndpoints.ListLots AdapterReachRealTreeTests: contract bypass … ListLots -> EggOperations through …IEggLotRepository
A member returning EggLot on IEggOperationsModule ModuleContractRealAssemblyTests
EggLotRepository removed from implementations CompatibilityExceptionRealTreeTests: undeclared EggLotRepository.* reads
The natural-key exception row renamed CompatibilityExceptionRealTreeTests: undeclared SeedFlockHistoryAsync read
The sweep's adapter row emptied AdapterReachRealTreeTests: undeclared LockDueEntriesAsync -> EggOperations

In the first attempt, two mutations (the sweep without its save, and the empty grade lookup) did not compile because a parameter became unread, so the tests ran against stale binaries and passed. I reran both with mutations that compile, and both turned red. The mutation script now refuses to run tests after a build error.

Cleanup. pstack:deslop ran on the branch diff before opening. It removed:

  • a repository comment that restated the method's signature;
  • a clause in the sweep's comment that narrated history.

Building with IDE0005 enabled then found three redundant using Cluckwork.Domain.Common; lines in new Application files, where the global using already covers them. Those were removed too. A /noslop review of the full diff found nothing more to remove.

Nothing user-visible changes, so this PR has no screenshots.

mforce added 5 commits October 2, 2026 21:10
The three FOR UPDATE reads over EggLots lock in (ProductionDate, Id)
order so confirm, sale void and daily-entry void queue instead of
deadlocking. Nothing failed when that order reversed: the full
integration suite stayed green with GetByIdsLockedAsync and
GetByDailyEntryLockedAsync ordered DESC. Each new test holds the later
lot, parks the request on it, and checks the earlier lot is already
held with FOR UPDATE NOWAIT. Reversing each order turns its test red.
…tract

Endpoints, both seeders and the daily-entry lock sweep reach Egg
Operations through IEggOperationsModule. SaleEndpoints reads grade names
through IEggGradeLookup, and feed and water usage read daily-entry
provenance through IDailyEntryLookup. The module only forwards to the
existing handlers and repositories; the sweep's lock loop moved into
LockDueDailyEntriesHandler unchanged. Commerce handlers are untouched:
the stock seam is #854's.

The ledger lists the contract, the four repositories as implementations,
five seeder reads as compatibility exceptions for #858, and the sweep's
new adapter row. Stale FK and edge reasons now name the ports.
EggOperationsModule copies entries, grades, lots and movements into
positional records, where two Guids or two counts can swap without a
compile error. Each test fails when two copied fields swap, when the
lot-existence check before a movement read goes, when the active-grade
list drops its farm, or when the daily-entry lookup swaps its key.
Adds the #853 decision record and its AGENTS.md paragraph: the
three-interface split, the egg lot lock order and its new pin, the
seeder rows left for #858, and the properties #854's stock seam must
keep.
Comment thread src/Cluckwork.Infrastructure/Jobs/DailyEntryLockSweep.cs Dismissed
@mforce

mforce commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

CodeQL check: one alert, dismissed as a false positive (#131).

cs/cleartext-storage-of-sensitive-information flagged DailyEntryLockSweep.cs:113. That is the sweep's existing summary log, Locked {Count} submitted entries … for account {AccountId}. The logged values are the same as on main. Only the count expression changed (lockedCount became pass.Locked.Count), which gave CodeQL a new fingerprint for the line.

The same statement on main (line 128) is dismissed alert #125. The same rule in this file is also dismissed as #1, #2, #114 and #124. AccountId is the tenant Guid carried in the JWT account_id claim: it is not a credential, a secret or PII. #131 is dismissed with that justification and can be reopened from the code-scanning page.

@mforce

mforce commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

Review of record: Codex gpt-6-astra (xhigh), round 1, at head 3f2584fe. Posted by the coordinator. Evidence paths named below are local to the review host.

PR #1028 review

Reviewed 3f2584fe1e52c7916e9d59144a4cdb3b39623dfd against 54a5f298, using the supplied diff and detached checkout. No delegation or GitHub writes. No product defect found at this head. One confirmed test defect needs correction; one minor test nit follows.

Finding

P2 · CONFIRMED · tests/Cluckwork.Api.IntegrationTests/EggLotLockOrderTests.cs:28 and :40

The confirm and sale-void tests never exercise the Id tiebreaker because their two lots have different production dates.

Concrete failure scenario: changing either GetAvailableFifoLockedAsync or GetByIdsLockedAsync to ORDER BY "ProductionDate", "Id" DESC makes it acquire same-date lots in the opposite order to the other paths, permitting a confirm/void deadlock, but all three new lock-order tests still pass. The daily-entry test exercises equal dates, but calls the third repository method, so it cannot protect these two reads.

I built and ran both mutations independently. Each returned 3 passed, 0 failed. Evidence: FIFO mutation, FIFO result, by-id mutation, by-id result.

Keep the different-date cases and add equal-date cases for both Commerce reads. Obtain canonical id order from PostgreSQL, and make the sale-void setup allocate from both lots. Each Id-only reversal should turn its test red. Until then, the decision record's claim that the tests pin the complete order on every path is too broad. This is a test defect, not a current product defect; production SQL remains correct.

Nits

P3 · CONFIRMED · tests/Cluckwork.Application.Tests/Eggs/EggOperationsModuleTests.cs:130

The movement-copy test cannot detect a dropped CreatedAtUtc: both its unpersisted movement and its expected record contain default.

Concrete failure scenario: replacing m.CreatedAtUtc with default in the module would return year-one timestamps on movement reads while this claimed field-complete test stays green. I built that mutation and all eight module tests passed. Mutation, result. Use a non-default source timestamp in a fixture or compare the GET response against a persisted timestamp. The current copy is correct; no current product defect.

Verification

dotnet build Cluckwork.sln passed with zero warnings and errors. I ran sg docker -c 'dotnet test Cluckwork.sln --no-build ...' against the unchanged head:

Project Passed Failed / skipped
Domain 495 0 / 0
Application 669 0 / 0
AppHost 10 0 / 0
Integration 1884 0 / 0

All 3,058 tests passed. Build, full run, counts for every suite listed in the PR.

This includes the submit/adjust/void races and 409s, ledger balance, sold floors, mortality rollback, audit provenance including DailyEntry.Submit, condition/restricted lots, tenancy, flock scope, feed/water provenance, seeders, and sweep tests. Existing invariant tests are unedited by this PR.

Every mutation compiled before its tests ran, and every source mutation was reverted:

Mutation Observed result
Reverse both ordering keys in FIFO read Confirm and sale-void lock tests red
Reverse both keys in by-id read Sale-void lock test red
Reverse both keys in daily-entry read Daily-entry lock test red
Reverse only Id, separately in FIFO and by-id reads All three lock tests green each time; finding above
Swap copied FarmId / HouseId Both entry-copy tests red
Replace copied movement timestamp with default Eight module tests green; nit above
Inject IEggLotRepository into StockEndpoints.ListLots Adapter guard red with the exact contract bypass
Nest all four requested entities inside EggLotDetails Contract guard red, identifying each entity
Put all four entities in a derived Result, then separately in a derived DailyEntryLockPass record Contract guard red in both cases, identifying each entity
Add unregistered reads of entries, grades, lots and movements in both Api and Infrastructure Compatibility guard red for both callers
Remove EggLotRepository from implementations Compatibility guard red
Remove the seeder natural-key exception Compatibility guard red
Remove the sweep handler's save Both sweep tests red

Mutation runner and exact transformations, exit-code ledger. Individual build logs, test logs, diffs and TRX results are alongside them.

Sweep and contracts

The sweep is behavior-equivalent except for the approved warning move. It retains 200 entries per account, Submitted and Date < farmToday - 7, date ordering, and one save for the batch, not one per entry. UnitOfWork.SaveChangesAsync directly delegates to the same scoped AppDbContext.SaveChangesAsync; no transaction or retry wrapper was introduced. Version conflicts still abort the batch, and success logs still follow the save.

The scheduled call remains behind DurableJobWorker's leader gate. The existing offline seeder drain remains unchanged. The sweep wrote no audit event before and writes none now; it still uses an account-resolved, unrestricted maintenance scope without invoking FlockScopeGuard. The audited handlers retain their actor checks, and seeders retain their Owner/persona resolution. There is no new system-actor privilege grant.

The locking probes use separate connections, a held later row, observed pg_blocking_pids, and FOR UPDATE NOWAIT on the earlier row. They do not infer ordering from elapsed sleeps. I found no new scheduling race under CI load; the shared 15-second observation deadline remains a timeout limit. The coverage gap above is independent of timing.

All returned DTO fields match their previous response projections. StockByGrade forwards unchanged, including its floor calculation. Both lookups preserve the original repository calls, arguments and tracking. Grade display names still include inactive grades. The filtered parent checks for daily-entry grades and lot movements remain intact.

The four implementation entries correctly name DailyEntryRepository, EggGradeRepository, EggLotRepository and EggInventoryMovementRepository. The five exception rows match the seeder reads, with Platform ownership and #858 deletion targets. All four corrected edge/FK reasons describe the new ports without widening permission. The natural-key AnyAsync preserves every predicate, including non-Voided status.

Matrix regeneration with CLUCKWORK_REGENERATE_MATRIX=1 produced no diff. A (24) becomes A (25) solely because DailyEntryLockSweep.LockDueEntriesAsync now resolves the module. The style build passes, with no new namespace segment shadowing the named egg types. The #854 amendment matches current confirm/void code: one farm-wide lock, planning over the same locked rows, allocation and restoration on those instances, correct movement references, and no seam-owned save. Commerce extraction remains deliberately outside this PR. Apart from the test-coverage overclaim above, the new decision record matches the implementation.

After restoration, the solution rebuilt cleanly, all 669 Application tests passed with matrix regeneration, and the three lock-order plus two sweep tests passed. git diff --exit-code and git status --porcelain=v1 were empty; HEAD remains the assigned commit.

7. CodeQL

Alert #131 is a false positive for this change, consistent with #125. At src/Cluckwork.Infrastructure/Jobs/DailyEntryLockSweep.cs:113, the value is the tenant's Guid selected from Accounts.Id, not a password, token, email or financial account number. The old line logged the same account id with lockedCount; the new line uses pass.Locked.Count. The move does not add sensitive content or change the account value logged. I inspected both alert records without changing their state.

Verdict: REQUEST CHANGES for the missing same-date tiebreaker tests; no product defect found.

…ment time

Astra round 1 on #1028. The confirm and sale-void lock-order tests used
lots on different dates, so reversing only the Id part of their ORDER BY
stayed green. Both now also run with two lots on one date, take the
canonical order from PostgreSQL, and the sale void draws from both lots.
Reversing the Id part of GetAvailableFifoLockedAsync,
GetByIdsLockedAsync or GetByDailyEntryLockedAsync turns its test red.

The module's movement copy test compared default timestamps on both
sides, so a dropped CreatedAtUtc passed. The fixture now carries a
stamped time.
@mforce

mforce commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

Response to Astra round 1 (at 3f2584fe). Both findings are fixed in 87ee7e99. No product code changed; only tests and the decision record's wording.

P2: lock-order tests missed the Id tiebreaker. Fixed.

  • ConfirmSale_HoldsTheEarlierLotWhileWaitingOnTheLater and VoidSale_HoldsTheEarlierSourceLotWhileWaitingOnTheLater are now theories. The daysApart: 5 row keeps the different-date case. The daysApart: 0 row puts both lots on one production date, where only Id orders them.
  • SeedTwoLotsAsync reads the canonical order from PostgreSQL (ORDER BY "ProductionDate", "Id" runs server-side), never from System.Guid comparison.
  • Both lots hold 30 eggs and the sale is 50. The sale-void setup asserts (0, 10) available after the confirm, which proves the sale drew from both lots in canonical order.
  • Each Id-only reversal (ORDER BY "ProductionDate", "Id" DESC) was run against a clean build, then reverted and rebuilt:
Read changed Red Green (expected)
GetAvailableFifoLockedAsync ConfirmSale_…(daysApart: 0) (the request must already hold the earlier lot while it waits on the later one); VoidSale_…(daysApart: 0) at its (0, 10) setup check, because the confirm then draws the other lot first both daysApart: 5 rows, VoidDailyEntry_…
GetByIdsLockedAsync VoidSale_…(daysApart: 0) (same lock message) the other 4
GetByDailyEntryLockedAsync VoidDailyEntry_HoldsTheEarlierLotWhileWaitingOnTheLater (same lock message) the other 4
  • On the restored code, the 5 lock-order tests passed 3 of 3 runs.
  • The decision record no longer claims more than the tests show. It now says the confirm and sale-void tests cover both parts of the order, that daily-entry lots always share a date, and that a whole-order or an Id-only reversal turns the matching test red.

P3: movement copy could not see a dropped CreatedAtUtc. Fixed. The fixture's movement now carries a stamped CreatedAtUtc (2026-09-21T08:30Z). It is set by reflection, the way a database read hands it back, which follows the precedent in SalesOrderCeilingTests. Replacing m.CreatedAtUtc with default in EggOperationsModule turns ListLotMovements_CopiesEveryField_UnknownLotIsNullWithoutAMovementRead red (Collections differ). With the copy restored, all 8 module tests pass.

Per the shared-host constraint, I ran only the targeted suites locally (lock order 5, module 8) and left the rest to CI on 87ee7e99. Local mutation evidence is in /home/mforce/.cluckwork-slices/853-logs/mut/ (r1-fifo-id-desc.txt, r2-byids-id-desc.txt, r3-bydailyentry-id-desc.txt, p3-movement-time-default.txt).

@mforce

mforce commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

Review of record: Codex gpt-6-astra (xhigh), round 2, at head 87ee7e99. Posted by the coordinator. Evidence paths named below are local to the review host.

PR #1028 review, round 2

Reviewed the fix from 3f2584fe to 87ee7e998a299d24fc5185858c326aeb7c003ff0. The diff changes two test files and the decision record; no product code changed. The prior P2 and P3 are resolved. No product defect or blocking finding remains.

Findings

None. The new equal-date cases detect the missing tiebreaker mutations, and the movement-copy test detects a dropped timestamp.

Nits

P3 · CONFIRMED · docs/decisions/853-egg-operations-contract.md:88; also tests/Cluckwork.Api.IntegrationTests/EggLotLockOrderTests.cs:109

The new wording incorrectly says PostgreSQL UUID ordering differs from the ordinary System.Guid ordering.

Concrete discrepancy: sorting the same UUID values with Guid.CompareTo and PostgreSQL's UUID comparator does not produce the different relative order claimed here. .NET compares the leading fields numerically with unsigned comparisons, then the remaining bytes; PostgreSQL compares the canonical UUID bytes. Those rules give the same ordering. Evidence: .NET 10 Guid.CompareTo, PostgreSQL 18.4 UUID parsing and comparison.

Keep the database-derived order, but describe it as using the database's canonical order and remove the claimed difference. The test itself orders correctly. Documentation/comment nit only; not a product defect and not blocking approval.

Mutation verification

I independently rebuilt and reran all three requested mutations. Every build passed before its tests ran; all mutations were reverted.

Mutation Result and causal failure
GetAvailableFifoLockedAsync: only Id DESC 2 red, 3 green. ConfirmSale_HoldsTheEarlierLotWhileWaitingOnTheLater(daysApart: 0) fails because the earlier lot is not already locked while the request waits on the later lot. The equal-date sale-void case also fails its setup assertion: expected (0, 10), actual (10, 0).
GetByIdsLockedAsync: only Id DESC 1 red, 4 green. Only VoidSale_HoldsTheEarlierSourceLotWhileWaitingOnTheLater(daysApart: 0) fails, at the earlier-lot lock assertion.
Movement copy: m.CreatedAtUtc becomes default 1 red, 7 green. ListLotMovements_CopiesEveryField_UnknownLotIsNullWithoutAMovementRead reports expected 2026-09-21 08:30:00 +00:00, actual 0001-01-01 00:00:00 +00:00.

The different-date cases stay green under both Id-only mutations, as expected. The FIFO mutation's confirm failure proves the locking assertion itself detects the regression; the additional sale-void setup failure is not being mistaken for coverage of the by-id read.

Evidence: FIFO result, by-id result, timestamp result, runner, exit-code ledger. Exact mutation diffs, build logs and TRX files are alongside them.

Determinism and clean runs

All five EggLotLockOrderTests cases passed three separate runs, including both equal-date cases each time: run 1, run 2, run 3. All eight module tests passed before and after the mutations. The solution builds passed with zero warnings and errors.

The tests obtain (ProductionDate, Id) order through server-side EF ordering before materialization. Both source lots hold 30 eggs, so a 50-egg sale must use both; (0, 10) also proves the sale-void setup allocated in canonical order. Random GUID values therefore cannot select the wrong held row.

The holder keeps an open transaction on the later lot. The HTTP request uses another connection, and pg_blocking_pids establishes that it is waiting before a third connection probes the earlier lot with FOR UPDATE NOWAIT. The holder is released only after that probe. No elapsed sleep substitutes for observing a lock, and there is no new scheduling race in the equal-date cases. The existing 15-second observation deadline remains a timeout bound under extreme CI starvation; these three runs are not a synthetic load test.

Adjacent lock paths and scope

The source census still finds three EggLot locked reads, all represented by the tests:

  • GetAvailableFifoLockedAsync: confirm, now covering different and equal dates. These fixtures use one grade.
  • GetByIdsLockedAsync: sale void, now covering both date cases. RecordEggLotMovementHandler also uses this read, but requests one lot, so there is no multi-row ordering decision on that path.
  • GetByDailyEntryLockedAsync: daily-entry void covers the equal-date order. Daily-entry adjustment calls the same repository method; it has no separate lock-observation case here, but introduces no additional SQL ordering implementation.

The tests do not establish that a multi-grade confirm takes one global locking statement rather than separate statements per grade. That is the acknowledged #854 test, outside #853's scope, and is not a finding against this PR. No additional EggLot locked-read implementation was found.

The timestamp fixture now supplies a fixed non-default value through the private setter, and the expected response retains that literal value. This closes the previous false-green assertion without changing production timestamp ownership.

7. CodeQL

The fix touches no sweep or logging code. Round 1's conclusion on alert #131 remains unchanged: the logged value is the existing tenant GUID, and the move does not add sensitive content. No GitHub alert or review state was changed.

All mutations are reverted, restored binaries were rebuilt, and the checkout is clean at 87ee7e998a299d24fc5185858c326aeb7c003ff0. This round reran the targeted suites, not the full backend suite already checked in round 1. No commits, pushes, GitHub comments or other checkouts were touched.

Verdict: APPROVE WITH NIT; both prior findings are resolved, with no product defect found.

The test and the decision record said PostgreSQL's uuid order differs
from System.Guid's. Astra's round 2 notes they agree. The reason the
test asks the database is that it owns the lock order.
@mforce

mforce commented Oct 2, 2026

Copy link
Copy Markdown
Owner Author

Astra round 2 P3 (wording) is fixed in 1c974acb. EggLotLockOrderTests.cs and docs/decisions/853-egg-operations-contract.md no longer claim that PostgreSQL's uuid order differs from System.Guid's. Both now give the actual reason: the test asks PostgreSQL, the database that applies the lock order, which lot is earlier, instead of relying on a C# comparison. Only a comment and the record's wording changed. The 5 lock-order tests pass locally, and CI is green on 1c974acb.

@mforce

mforce commented Oct 3, 2026

Copy link
Copy Markdown
Owner Author

Second review (independent), Claude Opus 5.5 (xhigh), at head 1c974acb. Requested by the owner because of the PR size; posted by the coordinator. Evidence paths named below are local to the review host.

PR #1028 second review (Opus), head 1c974acb

Reviewed 1c974acb026cb32df8925212e5c28243cc8f0656 against main c068b1e7, in the detached checkout
/home/mforce/.cluckwork-slices/review-1028-opus. I formed this view from the diff and the code before I
read the two Astra rounds. No delegation, no commits, no pushes, no GitHub writes. I reverted every mutation,
rebuilt the solution (0 warnings, 0 errors), and left git status empty at the assigned HEAD.

Evidence directory: /home/mforce/.cluckwork-slices/reviews/1028-opus-evidence/ (diffs and test logs for
each mutation, M1 to M7).

Product defects: none found. I have one test/doc finding (P2) and some nits.

Findings

F1 · P2 · CONFIRMED · test/doc, not product

Where: AGENTS.md:84 (sentence "EggLotLockOrderTests fails if any of the three reads changes order"),
tests/Cluckwork.Api.IntegrationTests/EggLotLockOrderTests.cs:110-113 (SeedTwoLotsAsync), and the reads
at src/Cluckwork.Infrastructure/Repositories/EggLotRepository.cs:56/78/99.

Defect: the lock-order tests catch a reversed order, as the decision record says. They cannot see a
dropped Id tie-breaker on the sale-void read, and they catch a dropped ORDER BY only by chance. So the
AGENTS.md claim is broader than the guard.

Why: lot Ids are random, and the test controls neither the Id order nor the insertion order.

  • In the equal-date row, "Id" = ANY(...) runs as a PK index scan. That scan already returns rows in Id
    order, which is the canonical order for equal dates.
  • In the different-date row, the earlier lot is always inserted first. Its Id is smaller only half the time.

So any plan order that matches "Id order" or "insertion order" passes the test whenever the coin falls the
right way.

Failure scenario: someone simplifies GetByIdsLockedAsync to ORDER BY "ProductionDate". Every test
stays green. In production, same-date lots then lock in plan order (heap order under a bitmap or seq scan),
while confirm locks them in (ProductionDate, Id) order. A sale void and a confirm on the same lots can
deadlock, and the rule this PR writes into AGENTS.md says the test would have caught it.

Evidence (5 lock-order tests per run; each mutation built clean first, then reverted):

Mutation Runs Result
M3: by-id read ORDER BY "ProductionDate" (drop Id) 3 5/5 green every run. Also 51/51 green over EggLotConcurrencyTests, EggLedgerTests, VoidSaleTests, TenantScopedLockTests, SaleAllocationPolicyTests, FlockScopeTests and EggLotLockOrderTests (m3-related-suites.log).
M1: by-id read, no ORDER BY 3 Red in runs 2 and 3 only, and only in VoidSale(daysApart: 5). The equal-date row is green every time (m1-run*.log).
M2: FIFO read, no ORDER BY 3 A different row goes red each run. P(all green) is about 1/4 per run (m2-run*.log).
M4: FIFO read ORDER BY "ProductionDate" 3 Same coin-flip pattern as M2 (m4-run*.log).

The daily-entry read is PLAUSIBLY in the same position: its lots also have random Ids and are inserted in
grade-line order. I did not run that mutation.

Fix, cheapest first:

  1. Narrow the AGENTS.md sentence to what holds: a reversed order, or a reversed Id part, turns the
    matching test red. The decision record (853-egg-operations-contract.md:89) already says only that.

  2. Harden the seed:

    • Give SeedEggLotAsync an optional lotId.
    • In the different-date row, give the earlier lot the larger Id (ask PostgreSQL to order two fresh
      Guids) and insert it second.
    • In the equal-date row, insert the larger-Id lot first.

    M1, M2 and M4 then fail every run.

  3. M3 cannot be caught behaviourally while the PK plan returns Id order. If the order must be pinned
    against a dropped key, add a plain text guard over the three SQL strings (ORDER BY "ProductionDate", "Id" directly before FOR UPDATE). The tenant-bypass allow-list keys these three reads by signature
    (I checked tenant-bypass-allowlist.json:118-130), so nothing else reads that text.

Checked and clean (no finding)

  • Forwarding parity. I diffed every endpoint before and after. Every forwarded call keeps the same
    arguments, cancellation token, Result mapping and round trips. This includes ListLotMovements, which
    still does the lot check and then the movement read, and stops on an unknown lot.
    SubmitDailyEntryAsync(id, accountId) forwards two Guids in the right order.
  • JSON parity, by reading. Every ToResponse is unchanged except for its input type. Every *Details
    field comes from the same entity property, and Grades keeps its order. The records drop only fields no
    response used: AccountId, the Standardize business-record timestamps and chronological list ordering #819 timestamps, and lot FlockId/Version. I did not diff live JSON.
  • Audit entity names. The audit entity-type constants (nameof(DailyEntry), nameof(EggGrade)) give
    the same strings as before.
  • LockDueDailyEntriesHandler against the old loop:
  • DI. The module, the lock handler and both lookups are scoped. The sweep resolves the module from a
    fresh scope per account, after tenant.Resolve. Every handler the module constructs has only scoped,
    side-effect-free constructor dependencies, so construction in a background scope cannot throw. The one
    registration method serves both serving and one-shot verbs, so the seeders resolve.
  • Tenant and flock scope (Add discovery guard for flock-scoped EF query filters #613, FlockScopeGuard's fail-open branch grants account-wide flock access to an unresolved actor #787).
    • Detail and list reads still go through the filtered GetReadOnlyAsync/ListAsync.
    • The movement read is still gated by the filtered lot read.
    • EggGradeLookup uses the tenant-filtered GetDisplayNamesAsync.
    • DailyEntryLookup wraps the same write-side bypass, still called after IFlockScopeGuard and still
      tracked.
    • The seeder's AnyAsync has the same predicate and the same filters as FindByNaturalKeyAsync (both
      keep the filters on).
  • Contract and reach guards. I tried leaks Astra did not:
    • M5: generic leaks on the two peer lookups (Task<IReadOnlyDictionary<Guid, IReadOnlyList<EggLot>>>,
      Task<KeyValuePair<Guid, EggGrade>?>). Red, with the exact type path.
    • M7: a repository interface as a property of a contract record. Red, through IRepository<EggLot, Guid>.
    • M6: adapter parameters IEnumerable<IEggLotRepository> and Func<IEggInventoryMovementRepository>.
      Red as contract bypasses.
  • Ledger.
    • The contract lists 26 types, which matches the record's breakdown.
    • All five exception rows match the seeder code: tables, IgnoreQueryFilters before tenant
      resolution, and child-first deletes.
    • All four rewritten reasons are accurate. No FlockManagement → EggOperations edge exists.
    • CI's matrix and compatibility tests are green.
  • Rule duties. No migration, config key, singleton, hosted service or user-visible change. [C] #514 slice 12: Commerce contract, consuming the stock seam #854 and
    [C] #514 slice 16: Platform composition, jobs, CLI and seeder conversion #858 carry amendments. CI is green on every check at this head.

Nits (P3, all CONFIRMED by reading)

  1. IDailyEntryRepository.FindByNaturalKeyAsync (IDailyEntryRepository.cs:28, DailyEntryRepository.cs:30)
    now has no caller in src/. Its only implementer outside the repository is the new test fake, which
    throws. Delete it.

  2. docs/decisions/855-inventory-contract.md:96 still says the feed and water record handlers keep
    IDailyEntryRepository. This PR moved them to IDailyEntryLookup.

  3. AGENTS.md:84 says "adapters call IEggOperationsModule, and peers call IEggGradeLookup and
    IDailyEntryLookup". But SaleEndpoints, an adapter, calls IEggGradeLookup. The decision record
    explains this; the canonical rule does not.

  4. IEggOperationsModule.LockSubmittedEntriesAsync (IEggOperationsModule.cs:40) is on the adapter
    surface. It does a flock-scoped bulk write with no FlockScopeGuard, no actor and no audit. Its comment
    does not say it is for the leader-gated sweep only, and an endpoint that called it would trip no guard.
    (Approved design; a comment is enough.)

  5. IDailyEntryLookup.FindIdForFlockScopedWriteAsync is now a peer-facing port.

    • It ignores both query filters and applies AccountId from its argument.
    • It applies no flock scope at all.
    • Its safety rests on the caller having run IFlockScopeGuard, and only a comment says so.

    This shape existed before, on the repository.

  6. #853 gets closed by this PR, but it carries no remainder note saying the stock seam moved to [C] #514 slice 12: Commerce contract, consuming the stock seam #854.
    AGENTS asks for one: "record the remainder on the issue". The [C] #514 slice 12: Commerce contract, consuming the stock seam #854 amendment holds the substance, so
    nothing is lost.

  7. docs/architecture.md:89-91 names the sweep as where the lock transition lives. The loop body now lives
    in LockDueDailyEntriesHandler.

Verification run

Run Result
Solution build at head (before and after mutations) 0 warnings, 0 errors
EggLotLockOrderTests, unchanged head (baseline) 5/5 pass
EggLotLockOrderTests, restored head 5/5 pass
Application EggOperationsModuleTests, ModuleContract*, AdapterReachRealTree*, CompatibilityException*, restored head 55/55 pass
CI on 1c974acb (gh pr checks) All checks pass; publish skipped as expected on a PR

I did not run the full integration suite. Astra ran it at 3f2584fe, and CI ran it at this head.

Is any finding a product defect? No. F1 is a test and documentation defect; the production SQL is
correct today.

Verdict: fix first. The fix is test and doc only (at minimum, narrow the AGENTS.md lock-order claim); no
product defect.

Opus review of #1028. The lock-order seeds used random lot Ids in
insertion order, so a dropped ORDER BY failed only by chance and a
dropped Id tie-breaker on the sale-void read never failed. The seeds now
insert the later lot first with fewer eggs, and on different dates give
it the smaller Id (PostgreSQL decides the Id order), so heap,
allocation-index and primary-key order disagree with the canonical
order. The daily-entry test seeds its entry and lots directly to pick
their Ids. Reversals, dropped tie-breakers and dropped ORDER BY clauses
on the three reads now fail every run, except the sale-void read's
tie-breaker: the confirm has already rewritten its lots in canonical
order. EggLotLockSqlTests pins the ORDER BY text of every FOR UPDATE
over EggLots, which covers it.

Also: drop the dead FindByNaturalKeyAsync, comment the sweep-only and
guard-first ports, and bring AGENTS.md, architecture.md and the #855
record up to date with the new ports.
@mforce

mforce commented Oct 3, 2026

Copy link
Copy Markdown
Owner Author

Response to the Opus review (at 1c974acb). Every item is fixed in ea81357b. No product behaviour changed. The commit touches tests, docs, comments and one deleted dead repository method.

F1 (P2): the lock-order pin caught a reversed order but not a dropped one. Fixed.

  • Seeds no longer leave the order to chance. SeedEggLotAsync takes an optional lotId and dailyEntryId. IdsInDatabaseOrderAsync asks PostgreSQL which of two fresh Guids is lower.
    • The canonically later lot is always inserted first and holds fewer eggs (30 against 40).
    • On different dates, the later lot also gets the lower Id.
    • So heap order, IX_EggLots_Allocation's (date, quantity) order and primary-key order all disagree with (ProductionDate, Id) wherever they can.
    • The daily-entry test now seeds its entry and both lots directly, so it chooses their Ids and insertion order.
    • A cross-check asserts that PostgreSQL's ORDER BY "ProductionDate", "Id" returns (earlier, later).
  • Mutation matrix. Four changes on each of the three reads. Each was built clean once, run 3 times, then reverted and rebuilt:
Read Id reversed Fully reversed Id tie-breaker dropped ORDER BY dropped
GetAvailableFifoLockedAsync RED ×3 ConfirmSale(0) RED ×3 ConfirmSale(0), ConfirmSale(5) RED ×3 ConfirmSale(0) RED ×3 ConfirmSale(0)
GetByIdsLockedAsync RED ×3 VoidSale(0) RED ×3 VoidSale(0), VoidSale(5) GREEN ×3 (behavioural), RED ×3 in EggLotLockSqlTests RED ×3 VoidSale(5)
GetByDailyEntryLockedAsync RED ×3 VoidDailyEntry RED ×3 VoidDailyEntry RED ×3 VoidDailyEntry RED ×3 VoidDailyEntry

Every red row failed with the request must already hold the earlier lot while it waits on the later one. The one exception is VoidSale(0) under the confirm-read mutations, which failed earlier, at its (0, 20) setup check, because the mutated confirm draws the other lot first.

  • Why the sale-void tie-breaker drop stays green behaviourally. EXPLAIN shows that read as a bitmap heap scan on PK_EggLots: LockRows → Sort ("ProductionDate") → Bitmap Heap Scan. Its rows come back in physical order. The confirm, which must run first, has just rewritten both lots, and it wrote their new versions in canonical order. So without the Id key the read still locks in canonical order. A test cannot change that without controlling where PostgreSQL places tuples, which is not deterministic. That is your option 3.
  • EggLotLockSqlTests (new, Application project). It walks src/ with Roslyn for every SQL string containing FOR UPDATE and "EggLots". Each must end ORDER BY "ProductionDate", "Id" FOR UPDATE, and the test fails below 3 statements. It fails when the tie-breaker is dropped from GetByIdsLockedAsync (3 of 3 runs, naming EggLotRepository.cs:72), when the FIFO read's ORDER BY is dropped, and when the daily-entry read's Id is reversed.
  • Claims narrowed to what the tests prove. AGENTS.md:84 and the [C] #514 slice 11: Egg Operations contract and the opaque stock seam #853 decision record now say what each test catches, and name the one change only the SQL guard sees.
  • Targeted runs at this head: EggLotLockOrderTests + FeedUsageTests + WaterUsageTests, 25 of 25 on 3 runs. Application tests filtered to Eggs, Architecture, TenantBypass and Documentation, 406 of 406. The doc-walking guards, 4 of 4. The full suite is left to CI.

P3 nits, all seven fixed:

  1. FindByNaturalKeyAsync is deleted from IDailyEntryRepository, DailyEntryRepository and the throwing test fake. Its comment on why voided entries are excluded moved to the write-side lookup it also describes. CompatibilityExceptionRealTreeTests stays green.
  2. 855-inventory-contract.md:96 now says the feed and water record handlers read provenance through IDailyEntryLookup since [C] #514 slice 11: Egg Operations contract and the opaque stock seam #853.
  3. AGENTS.md:84 now says that SaleEndpoints, an adapter, also takes IEggGradeLookup.
  4. IEggOperationsModule.LockSubmittedEntriesAsync now notes that it is for the leader-gated DailyEntryLockSweep only, and runs no flock-scope guard, needs no actor and writes no audit row.
  5. IDailyEntryLookup.FindIdForFlockScopedWriteAsync now says the caller must run IFlockScopeGuard first, because the read ignores both query filters and applies only the given AccountId.
  6. The remainder note is on [C] #514 slice 11: Egg Operations contract and the opaque stock seam #853: [C] #514 slice 11: Egg Operations contract and the opaque stock seam #853 (comment). It links the [C] #514 slice 12: Commerce contract, consuming the stock seam #854 amendment.
  7. docs/architecture.md names LockDueDailyEntriesHandler as where the lock loop lives, with the sweep picking each farm's cutoff.

Local evidence is in /home/mforce/.cluckwork-slices/853-logs/:

  • mut3/ holds the 12 mutation logs and summary.txt.
  • mut/s1-byids-dropid.txt, s2-fifo-noorder.txt and s3-daily-iddesc.txt are the SQL-guard mutations.
  • byids-plan.txt is the EXPLAIN output.

No further review round will be triggered on this PR.

@mforce
mforce merged commit d6dbd47 into main Oct 3, 2026
19 checks passed
@mforce
mforce deleted the refactor/853-egg-operations-contract branch October 3, 2026 04:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C] #514 slice 11: Egg Operations contract and the opaque stock seam

2 participants