Skip to content

test: p2p_node_network_limited.py --v2transport intermittently disconnects during connect_nodes #7288

Description

@thepastaclaw

Summary

linux64_tsan-test / Test source intermittently fails in p2p_node_network_limited.py --v2transport with AssertionError: Error: peer disconnected. This is not caused by PR-specific code in dashpay/dash#7230; the same head SHA passed on rerun without any branch changes.

Evidence

Failure mode

The failure happens here:

File "test/functional/p2p_node_network_limited.py", line 83, in run_test
    self.connect_nodes(0, 2)
...
AssertionError: Error: peer disconnected

Combined logs show node 0 immediately disconnecting node 2 after node 2 requests a block below the NODE_NETWORK_LIMITED threshold:

ProcessGetBlockData [net] Ignore block request below NODE_NETWORK_LIMITED threshold, disconnect peer=2

connect_nodes() is still waiting for the outbound peer to stay connected long enough to exchange a pong, so the helper fails with peer disconnected.

Diagnosis

This looks timing-sensitive / transport-sensitive rather than PR-specific:

  • PR #7230 only changes src/node/interfaces.cpp and src/wallet/wallet.cpp.
  • The failing test is test/functional/p2p_node_network_limited.py.
  • The exact same PR head passed on rerun, so there is no deterministic wallet-side regression here.

The likely issue is that the test currently assumes connect_nodes(0, 2) will remain connected long enough for the helper handshake, but under TSAN + --v2transport the pruned node can disconnect node 2 quickly enough that the helper trips first.

Reproduction ideas

I have not reproduced this locally outside CI yet. The closest reproduction path is to loop the test under a slow / TSAN-like environment:

python3 test/functional/test_runner.py p2p_node_network_limited.py --v2transport

or repeatedly rerun the TSAN functional shard in CI until the timing window appears.

Suggested direction

Harden the test so it does not rely on connect_nodes() succeeding when the scenario itself can legitimately trigger a fast disconnect. For example, make the unsynced-node phase explicitly tolerate the disconnect and assert the expected postcondition (node2 stays at height 0) without requiring a stable pong handshake first.

Activity

  1. thepastaclaw commented on Jul 11, 2026

    @thepastaclaw
    CollaboratorAuthor

    New occurrence on PR #7418, exact head 4f129b609e3aed8c55eb9006b06982fa1c88d878:

    • TSAN job: https://github.com/dashpay/dash/actions/runs/29144397638/job/86524311711
    • p2p_node_network_limited.py --v2transport failed on all 3 attempts at connect_nodes(0, 2) with AssertionError: Error: peer disconnected.
    • The combined log confirms the same race described here: node 0 processes the below-NODE_NETWORK_LIMITED block request and logs disconnect peer=2 before connect_nodes() observes the expected pong.
    • PR fix(net): bound signing message vector intake #7418 changes only src/llmq/net_signing.cpp, src/llmq/net_signing.h, src/llmq/signing_shares.h, and src/test/llmq_utils_tests.cpp, so there is no overlap with the failing test or connection helper.

    This is another pre-existing timing-flake occurrence; the PR branch was left unchanged.

  2. thepastaclaw commented on Jul 18, 2026

    @thepastaclaw
    CollaboratorAuthor

    New occurrence on #7401 at exact head 45821d0:

    • Failing TSAN job: https://github.com/dashpay/dash/actions/runs/29400920523/job/87307391340
    • p2p_node_network_limited.py --v2transport failed all three attempts in connect_nodes(0, 2) with AssertionError: Error: peer disconnected.
    • The combined log shows the same race documented here: node 0 processes the below-NODE_NETWORK_LIMITED block request and disconnects peer 2 before connect_nodes observes the pong.
    • The exact-head PR diff is limited to DKG queueing/validation files and focused DKG tests; it does not touch this test, connect_nodes, or NODE_NETWORK_LIMITED handling.
    • The same test passed on this exact head in both linux64-test and linux64_ubsan-test.

    This is another pre-existing timing-sensitive occurrence. The PR branch was left unchanged.

  3. thepastaclaw commented on Jul 24, 2026

    @thepastaclaw
    CollaboratorAuthor

    New occurrence on PR #7433 at exact head 5b499d2089043fdf9be9e92893e8c40fd6a61a98:

    • Failing TSAN job: https://github.com/dashpay/dash/actions/runs/30121777504/job/89581303737
    • p2p_node_network_limited.py --v2transport failed in connect_nodes(0, 2) with AssertionError: Error: peer disconnected.
    • The uploaded combined log confirms the same established race: node 0 correctly logs Ignore block request below NODE_NETWORK_LIMITED threshold, disconnect peer=2, then connect_nodes() observes the disconnect before its expected pong.
    • There is no ThreadSanitizer race report. The PR does not change this test, connect_nodes, or the NODE_NETWORK_LIMITED handling; its test_node.py delta is limited to expected process-return-code handling. Non-TSAN lanes passed the test on the same head.

    The branch was left unchanged; PR classification: #7433 (comment)

  4. thepastaclaw commented on Jul 29, 2026

    @thepastaclaw
    CollaboratorAuthor

    New occurrence on PR #7496 at exact head 9bc7ed53296f3d98c3fbbd8f6991a04693702c3b:

    • Failing TSAN job: https://github.com/dashpay/dash/actions/runs/30487741775/job/90699995346
    • p2p_node_network_limited.py --v2transport failed all three attempts at connect_nodes(0, 2) with AssertionError: Error: peer disconnected.
    • The combined log shows the same established race: node 0 correctly logs Ignore block request below NODE_NETWORK_LIMITED threshold, disconnect peer=0; connect_nodes() then observes that disconnect before the expected pong. The unexpected pruned-node warning during shutdown is secondary to the original assertion.
    • There is no ThreadSanitizer race report. PR ci: reject @mentions in pull request descriptions #7496 changes only .github/workflows/check_pr_description_mentions.py, .github/workflows/semantic-pull-request.yml, and .github/workflows/test_check_pr_description_mentions.py; it does not touch this test, connect_nodes, networking, or NODE_NETWORK_LIMITED handling.
    • The same test passed in the non-TSAN, UBSAN, SQLite, no-wallet, and multiprocess test lanes on this exact head.

    This is another pre-existing timing-sensitive occurrence. The PR branch was left unchanged.

  5. added a commit that references this issue on Aug 3, 2026
    f1dde51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions