Skip to content

fix: reject renewal of lost and expired cache leases - #573

Closed
niemyjski wants to merge 4 commits into
mainfrom
fix/cache-lease-ownership-20260922
Closed

niemyjski wants to merge 4 commits into
mainfrom
fix/cache-lease-ownership-20260922

Conversation

@niemyjski

@niemyjski niemyjski commented Sep 22, 2026

Copy link
Copy Markdown
Member

Summary

Related consistency dependency for FoundatioFx/Foundatio.Repositories#307 and #327. Current head: c4fbcd4c173969b5560f88954c54ae62307f7a95.

Three lease defects are addressed:

  1. CacheLockProvider.RenewAsync now checks the compare-and-replace result and throws LockException when ownership was lost. Failed renewal does not increment the wrapper's renewal count.
  2. Expired or released in-memory entries cannot be resurrected by conditional renewal.
  3. Conditional cache replacement/removal now commits against the exact compared entry. A retried dictionary update factory previously retained a successful-comparison flag from a stale invocation and could report success after another owner replaced the entry. The new implementation uses compare-and-swap/exact-entry removal without mutating the comparison entry first.

Another owner's lease is not renewed or removed, and normal successful renewal remains supported. Documentation identifies this intentional behavior correction and explicitly distinguishes leases from storage-enforced fencing.

Commits

Commit Change
2f1c51e0 Surface lost ownership and reject expired/released renewal; regression tests and documentation.
b2117439 Make conditional cache replacement/removal compare-and-swap safe; deterministic concurrent replacement tests.
c4fbcd4c Make the existing throttle-period test use a controlled clock rather than assuming a wall-clock period cannot change mid-test. No throttle production behavior changed.

Executed validation

Run 35795463344: initial implementation tests were 3 failed / 1 passed before repair; 425 lock/cache cases passed, none failed or skipped, afterward. Release build compiled .NET8/.NET10 libraries with zero warnings/errors at that revision.

Run 35799107410 generated the exact two follow-up commits:

  • Both controlled concurrent-replacement regressions failed without the CAS repair.
  • Final selected lock/cache suite: 427 passed, 0 failed, 0 skipped; tests ran on .NET10.
  • Release solution build: 0 errors, with 1 existing ASPIRE010 sample-project configuration warning. The warning was not suppressed or presented as a clean-zero-warning result.
  • Changed-file whitespace verification and git diff --check passed.
  • Artifact 10725336070 contains the test/build logs, exact commit IDs, patch and bundle.

The first CAS validation had 426 passes and one failure in the old wall-clock throttle test. Its trace straddled a two-second throttle window, so the test incorrectly expected the next call to be rejected. The clock-controlled replacement still verifies 25 accepted calls, rejection within the exhausted period, and acceptance after advancing to the next period.

No audit workflow, release configuration, dependency bump, or publishing change is in this PR. The branch was advanced without force-pushing. Final-head ordinary CI is separate from the above inspected candidate evidence.

Integration and remaining boundaries

Repositories still references released Foundatio 13.0.4. This PR must be merged/released and explicitly consumed before repository migrations gain the provider repair. Opening or updating this PR does not change consumers' installed package.

These changes do not revoke already-dispatched external operations, implement a migration fencing token, or certify every unrelated cache mutation race. Migration journal/takeover protections remain separate release gates on the repository PRs. Custom cache and lock providers must honor equivalent conditional ownership semantics.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 22, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-23T02:05:33.182489Z e2f4d12 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2f1c51e015

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/Foundatio/Caching/InMemoryCacheClient.cs Outdated
Comment thread src/Foundatio/Lock/CacheLockProvider.cs
Comment thread docs/guide/locks.md

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c4fbcd4c17

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/Foundatio/Caching/InMemoryCacheClient.cs

Copy link
Copy Markdown
Member Author

Sequential follow-up for Repositories #307/#327: inspected this PR at c4fbcd4c173969b5560f88954c54ae62307f7a95, including the checked renewal and exact-entry CAS repairs. Downloaded artifact 10725336070 from run 35799107410: both controlled CAS regressions fail on the unpatched code; final lock/cache run reports 427 passed, 0 failed, 0 skipped; the build reports 0 errors and the documented existing ASPIRE010 warning.

This existing implementation is the upstream provider repair for step 1 of FoundatioFx/Foundatio.Repositories#307 (comment). It is not being duplicated. Downstream consumption remains open: Repositories still references 13.0.4; no invented package version or release is being substituted. The migration-side heartbeat, lost-ownership behavior, and outstanding-request fencing are separate checks and are not certified by these provider tests.

Copy link
Copy Markdown
Member Author

Repository migration dependency — sequential gate 1

This PR is the first release gate for Foundatio.Repositories #307/#327.

The latest commit e2f4d123 additionally closes the expiry-at-CAS window: if a conditional renewal's value comparison pauses across the lease deadline, a successful dictionary CAS is rechecked against the prior entry's expiry and the attempted revival is removed/fails. It also documents LockException on both public renewal abstractions and records the lost-lease rule in the Foundatio skill.

Repositories must not consume this behavior until this PR is merged and released. Even after consumption, lease renewal is coordination—not storage fencing—so the migration PRs still separately need single-dispatch/task-lineage and stale-controller protections.

I am holding the downstream sequence on this exact head's CI before moving to the next runtime gate.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e2f4d12392

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1133 to +1135
if (entry.IsExpired)
{
if (((ICollection<KeyValuePair<string, CacheEntry>>)_memory).Remove(new KeyValuePair<string, CacheEntry>(key, replacement)))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Make expiry validation part of the atomic mutation

When two renewals for the same lease overlap at the expiry boundary, this post-CAS cleanup can still leave the expired lease renewed: the first call installs replacement after the old entry expires and pauses before this check, while the second call observes that fresh replacement, successfully replaces it again, and returns true; the first call then detects the old expiry but cannot remove its no-longer-current replacement. The new post-CAS removal is fresh evidence that the previously reported boundary race remains possible, so expiry must be enforced as part of the mutation rather than by best-effort cleanup afterward.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #570 (d207f44). ReplaceIfEqualAsync now checks expiry inside the same UpdateEntry callback that publishes the change, after the value comparison, so expiry is part of the compare-and-swap rather than a cleanup step afterward. Both the overlapping-renewal case and the paused-comparison case are covered by RemoveIfEqualAsync_WhenEntryExpiresDuringComparison_ReturnsFalse.

Comment on lines +1115 to +1117
var replacement = CreateEntry(value, expiresAt);
if (replacement is null)
return false;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Defer constructing the replacement until the value matches

When the key is absent, expired, or contains a mismatched value, constructing the replacement here now clones the supplied value and runs its size calculator before the method checks whether replacement is permitted. Consequently an unclonable value, a throwing custom calculator, or an oversized value with ShouldThrowOnMaxEntrySizeExceeded can now throw where ReplaceIfEqualAsync previously returned false without inspecting the replacement; perform replacement construction only after the expected value has matched.

AGENTS.md reference: AGENTS.md:L162-L166

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #570. The replacement is built lazily, only after the expected value matches (and now only if the entry hasn't expired). Covered by ReplaceIfEqualAsync_WithMismatchedOversizedValue_ReturnsFalseWithoutThrowing.

niemyjski added a commit that referenced this pull request Sep 23, 2026
Folds in the lock-ownership fixes from #573 on top of UpdateEntry:
- ReplaceIfEqualAsync/RemoveIfEqualAsync treat an expired entry as a miss, checked
  after the value comparison so a lease that expires mid-comparison is not revived
- CacheLockProvider.RenewAsync throws LockException when renewal fails instead of
  silently reporting success; documented on ILockProvider and ILock
- Multi-resource TryAcquireAsync releases what it holds and returns null when an
  earlier lock is lost during acquisition
- Clock-controlled throttle test, docs, and skill gotcha
@niemyjski

Copy link
Copy Markdown
Member Author

Superseded by #570, which now includes this PR's lock-ownership fixes, rebuilt on #570's single UpdateEntry write path:

  • CacheLockProvider.RenewAsync throws LockException when renewal fails (documented on ILockProvider/ILock, plus docs and skill)
  • ReplaceIfEqualAsync/RemoveIfEqualAsync treat an expired entry as a miss. Expiry is checked inside the compare-and-swap callback after the value comparison, which also resolves the two open Codex threads here.
  • Multi-resource TryAcquireAsync releases what it holds and returns null when an earlier lock is lost (new; the throw would otherwise have leaked locks)
  • Clock-controlled throttle test

Both PRs rewrote the same ReplaceIfEqualAsync/RemoveIfEqualAsync code, so they couldn't both merge. The Foundatio.Repositories release gate for #307/#327 should now track #570. The branch is left in place.

@niemyjski niemyjski closed this Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant