Sweep for emancipated connections while callers are parked - #4529
Open
mdaigle wants to merge 1 commit into
Open
Conversation
ChannelDbConnectionPool only reclaimed emancipated connections inline, on the caller's own thread, immediately before parking on the idle channel. A connection becomes emancipated when its owning SqlConnection is collected without being closed, and that can only be observed after a GC. If the GC lands after the caller has already parked, nothing sweeps again: every caller is blocked on the channel, so the pool stays saturated until an unrelated caller arrives to run its own inline sweep. With all callers parked, that never happens and they all fail on their connect timeout. The sweep cannot simply be retried on the parked caller's thread. Channels guarantee FIFO delivery to ReadAsync callers, which the pool relies on for fairness, so a caller that cancelled its read to re-sweep would rejoin at the back of the queue behind callers that arrived later. The trigger has to come from off-thread. PoolReclaimer adds a demand-driven timer for that. Callers register around their parked wait, the timer arms on the first registration and disarms on the last, so a pool that is not blocking pays nothing beyond the ~100 bytes of a disarmed timer, which is not in the timer queue's list and does not lengthen any tick. It is separate from PoolPruner rather than folded into it: a merged timer would have to run at the faster of the two cadences, and the prune interval is derived from the idle timeout and stretches to 288s at the default, so merging would multiply prune-driven ticks by ~28x. The reclaimer is built for every pool configuration, unlike the pruner, which is null for a fixed-size pool or a zero idle timeout. A connection can leak in any configuration, and a fixed-size pool is where a leaked slot hurts most. The sweep runs one-shot and re-arms at the end of each callback so a slow sweep cannot overlap the next, sweeps outside its lock because reclamation can make server round trips, and swallows exceptions because a throwing timer callback would tear down the process. The timer is created via ADP.UnsafeCreateTimer so it does not capture the execution context of whichever caller happens to park first, which would otherwise pin that caller's async locals for the lifetime of the pool. The one-second cadence is far tighter than the legacy pool's background reclaim, which rides a randomized 2-4 minute cleanup wait and is too slow to rescue a caller inside a 15 second connect timeout. Sweeping only while callers are parked is what makes the tighter cadence affordable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a demand-driven background reclaim timer to ChannelDbConnectionPool so emancipated (GC-orphaned) connections can be swept and returned to the idle channel while all callers are blocked waiting, preventing pool saturation/timeouts in the “everyone parked” scenario described in #4490.
Changes:
- Introduces
PoolReclaimer, a one-shot re-arming timer that sweeps for emancipated connections only while callers are parked. - Wires reclaim registration around the idle-channel wait and disposes the reclaimer during pool shutdown before draining.
- Adds
ChannelDbConnectionPoolReclaimTimerTestunit tests to cover timer arming/disarming and an end-to-end “GC after parking” wake-up scenario.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/ChannelDbConnectionPoolReclaimTimerTest.cs |
New unit tests validating PoolReclaimer behavior and the end-to-end “park then GC then wake” scenario. |
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/ConnectionPool/PoolReclaimer.cs |
New demand-driven timer component that runs background sweeps while callers are parked. |
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/ConnectionPool/ChannelDbConnectionPool.cs |
Integrates PoolReclaimer creation, parked-wait registration, internalizes ReclaimEmancipatedConnections, and disposes the reclaimer during shutdown. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+119
to
+124
| ThreadPool.GetMinThreads(out int workerThreads, out int completionPortThreads); | ||
| int desired = Environment.ProcessorCount * 4; | ||
| if (workerThreads < desired) | ||
| { | ||
| ThreadPool.SetMinThreads(desired, completionPortThreads); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #4490. Base is
dev/automation/channel-pool-v2-parity, so the diff here is only the timer work and can be compared against the inline-only reclaim in #4490.The gap
#4490 reclaims emancipated connections inline, on the caller's own thread, right before it parks on the idle channel. Emancipation only becomes observable after a GC. If that GC lands after the caller has parked, nothing sweeps again: every caller is blocked on the channel, so the pool stays saturated until an unrelated caller arrives and runs its own inline sweep. With all callers parked, that never happens and they all fail on their connect timeout.
For comparison,
WaitHandleDbConnectionPoolsweeps in four places, one of which is a background threadpool callback. The channel pool has one.Why not sweep on the parked caller's thread
Channels guarantee FIFO delivery to
ReadAsynccallers, which the pool relies on for fairness. A caller that cancelled its read to re-sweep would rejoin at the back of the queue behind callers that arrived later. So the legacy pattern (bounded wait, wake, re-sweep, re-wait) does not port, and the trigger has to come from off-thread.Why not fold this into
PoolPrunerA merged timer has to run at the faster of the two cadences. The prune interval is derived from the idle timeout and stretches to 288s at the default, so a 1s maintenance tick would multiply prune-driven ticks by ~28x. Timer count is nearly free; armed ticks are what cost. A disarmed timer is ~100 bytes and is not in the timer queue's list, so it does not lengthen any tick.
PoolPruneralso has three coverage holes for this purpose: it is null whenMinPoolSize >= MaxPoolSizeorIdleTimeout == 0, and it disarms whenCount <= MinPoolSize. Those are exactly the configurations where a leaked slot hurts most.What this adds
PoolReclaimer: a demand-driven timer. Callers register around their parked wait, it arms on the first registration and disarms on the last, so a pool that is not blocking pays nothing. It is constructed for every pool configuration.ADP.UnsafeCreateTimerso it does not capture the execution context of whichever caller happens to park first, which would otherwise pin that caller's async locals for the lifetime of the pool.Shutdown, before the drain, so an in-flight sweep cannot route a connection back into a channel the drain has already passed.Cadence
1s. The legacy pool's background reclaim rides a randomized 2-4 minute cleanup wait, which is too slow to rescue a caller inside a 15s connect timeout; legacy effectively depends on its inline sweeps instead. Sweeping only while callers are parked is what makes the tighter cadence affordable, since an idle pool does no work at all.
Tests
10 unit tests in
ChannelDbConnectionPoolReclaimTimerTest, covering construction across pool configurations, arm/disarm transitions, re-arm after a full drain, the disarmed no-op path, shutdown disposal, and an end-to-end test where a caller parks, the owner is collected only afterward, and the sweep wakes it.The end-to-end test was verified to genuinely fail without the fix (times out on the full connect timeout).
Validation
ConnectionPool+SimulatedServerTestsfilter, before and after: identical results apart from the 10 new passing tests. The one failure (IntegratedAuthConnectionTest, SSPI) and theServerDoesNotRoutehang both reproduce with this change reverted and are pre-existing on this machine. Only net9.0 was built and tested locally.Deliberately out of scope
Left out to keep the comparison diff isolated, worth follow-ups:
MaxPoolSize. Today a leaky app onMaxPoolSize=100pays 99 extra physical connects before the first sweep.Clear(). It only drains the idle channel, so a leaked connection keeps its slot.SqlClientDiagnostics.Metrics.ReclaimedConnectionRequest()in the channel pool's sweep.number-of-reclaimed-connectionscurrently reads 0 under pool V2.MaxPoolSizeparameter onConnectionPoolChurnRunnerto guard hot-path allocations, and a leak-recovery runner measuring time-to-open atMaxPoolSizeafter leaking connections.