test: harden the flakiest unit test and the contract test harness download - #386
Draft
kinyoklion wants to merge 1 commit into
Draft
test: harden the flakiest unit test and the contract test harness download#386kinyoklion wants to merge 1 commit into
kinyoklion wants to merge 1 commit into
Conversation
…on harness download errors Co-Authored-By: rlamb@launchdarkly.com <4955475+kinyoklion@users.noreply.github.com>
Contributor
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requirements
Related issues
Comes out of a review of every
Build and Testrun (PR +main) over the last three months, including all attempts of re-run builds. The FDv2OutOfMemoryErrorflakes found by that review are being fixed separately in #384 and are deliberately untouched here.Describe the solution you've provided
Two independent flake sources, both outside the SDK's production code.
StateDebounceManagerTest.closeCancelsPendingTimerscheduled a debounce timer, slept for a fraction of the window, and expectedclose()to win the race. On a loaded runner the sleep overshoots the window, the timer fires first, and the test fails. It now drives the timer through the existingManualTaskExecutor, so cancellation is observed directly rather than inferred from wall-clock ordering, and additionally asserts that the scheduled task really was cancelled.resetCancelsPendingTimerhad the identical shape and gets the same treatment. Two sibling tests were already converted this way in earlier fixes; these were missed.The contract test step fetched the harness downloader with
curl -s ... | sh. Beyond having no retry, that pipeline is silently fatal in the wrong direction: a failed fetch feedsshan empty script, which exits 0, so the contract tests would be skipped and the build would go green. The downloader is now fetched to a file with--failand retries, and run from there, so a fetch failure fails the build. Both suites share that one copy, taken from the branch whose downloader retries the release download — the script is version-agnostic,VERSIONstill selects which harness binary each suite fetches, which is what fixes the observedDownload failedfailures on the v3 run.Describe alternatives you've considered
Retrying the whole contract-test step would also mask genuine failures, and re-running the suite after the emulator has booted is expensive. Pinning
VERSION=v2to an exact release (as the v3 line already does) is a separate recommendation: the floating tag makes contract results non-reproducible and has already produced a red-then-green build with no code change, but changing it is a process decision rather than a flake fix, so it is not part of this PR.Additional context
Verified with
:launchdarkly-android-client-sdk:test(full unit test suite) plus repeated--rerun-tasksruns of the changed class, and:launchdarkly-android-client-sdk:lint. The Makefile change was exercised both ways: a successful download and run of the harness, and a deliberately bad downloader URL, which now aborts the target instead of quietly skipping the tests.Link to Devin session: https://app.devin.ai/sessions/fb9541c6fa994866a832460742c89682
Requested by: @kinyoklion