Skip to content

fix(captcha): move happy-dom solve off the proxy event loop into workers - #55

Merged
TriDefender merged 5 commits into
TriDefender:masterfrom
Jasmine-Lee-2026:fix/captcha-worker-solve
Sep 27, 2026
Merged

TriDefender merged 5 commits into
TriDefender:masterfrom
Jasmine-Lee-2026:fix/captcha-worker-solve

Conversation

@Jasmine-Lee-2026

@Jasmine-Lee-2026 Jasmine-Lee-2026 commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Closes #54

PR draft: fix(captcha): move happy-dom solve off the proxy event loop

What this PR does

On the start-plan path, captcha solving (happy-dom + sync XHR via Atomics.wait) runs on the proxy's main thread. After the token pool drains during idle, the refill burst freezes the event loop repeatedly for tens of seconds; every connection hangs until restart (reproduction steps and log evidence in the companion issue).

This PR moves each solve into a dedicated worker thread. The main thread only dispatches the job and waits on the result with a hard timeout. A hung solve can no longer affect anything outside its own worker.

Changes (3 commits, mitigation and structural fix separated for easy review)

1. fix(captcha): cap main-thread blocking windows in happy solver (mitigation)

  • captcha-happy.ts: syncFetchBlocking timeout 30s -> 12s (env CAPTCHA_SYNC_FETCH_TIMEOUT_MS)
  • captcha-happy.ts: solveTraceless overall deadline 30s -> 20s (env CAPTCHA_SOLVE_TIMEOUT_MS)

2. fix(captcha): bound config fetch with AbortSignal and negative cache (mitigation)

  • captcha.ts: fetchCaptchaConfig gets a 5s AbortSignal timeout (env CAPTCHA_CONFIG_TIMEOUT_MS)
  • 15s negative cache after a failed fetch so a broken configs endpoint cannot repeatedly wedge request handling

3. fix(captcha): move happy-dom solve off the proxy event loop into workers (structural fix)

  • New src/proxy/captcha-worker-entry.ts: worker entry running happy-dom solveTraceless over postMessage (ESM)
  • New scripts/build-fork-worker.ts: pre-bundles the entry into a self-contained captcha-worker-entry.bundle.js (bun build --target=bun --bundle --format=esm), embedded into the compiled binary via import ... with { type: "file" }. Under bun build --compile, new Worker(new URL()) with a file path does not work; this asset-based approach is the verified alternative.
  • Rewrite src/proxy/captcha-solver.ts: worker-per-solve scheduler. Each solve gets its own worker with a 20s hard-terminate deadline; spawn/exit/error all have fallback paths. The existing concurrency-control API is kept as no-op stubs for compatibility.
  • The bundle is about 1.7MB (self-contained happy-dom + undici); binary size grows accordingly.

Compatibility notes

  • No changes to captcha-pool.ts or the token acquisition flow; setCaptchaSolverConcurrency / shutdownCaptchaSolver signatures unchanged.
  • All new timeouts are env-overridable; defaults are the values above.
  • The webui commit (4f9b6e2) touches none of the captcha files; rebase is clean.

Verification

  • bun x tsc --noEmit clean
  • Full bun test: 885 pass / 0 fail (includes 28 captcha-related tests)
  • Windows x64 + bun 1.4.2 compiled binary, live testing:
    • Real request completes the full worker decode path, HTTP 200
    • 3 concurrent requests all 200 in 1.8-1.9s
    • serve debug error log empty (the same scenario always produced main-thread stall records before the fix)
    • Wake-up after >= 15 minutes idle: no recurrence over multiple days
  • The bundling script is included in scripts/; run bun run scripts/build-fork-worker.ts before bun run build to reproduce the artifact.

Known trade-offs

  • worker-per-solve pays one spawn per solve (measured at ms level, negligible vs. the previous 8-way serialized event-loop freeze).
  • If a persistent worker pool is preferred instead, the scheduler interface is isolated and can be swapped without touching callers.

(Chinese translation attached for reference: upstream-pr-draft.zh.md)

Update (bundle no longer committed)

The worker bundle is now a gitignored build artifact, matching the upstream convention used for the Android server bundle: package.json wires scripts/build-fork-worker.ts as a prebuild hook, so �un run build always regenerates it before compiling. The diff is now ~250 lines of real code instead of ~50k lines of generated output (commit bd07d6e).

- syncFetchBlocking timeout 30s -> 12s (CAPTCHA_SYNC_FETCH_TIMEOUT_MS)
- solveTraceless overall timeout 30s -> 20s (CAPTCHA_SOLVE_TIMEOUT_MS)

Mitigation only: these bounds reduce how long a single solve can stall
the proxy event loop, they do not remove the stall.
- fetchCaptchaConfig: 5s abort timeout (CAPTCHA_CONFIG_TIMEOUT_MS)
- 15s negative cache after a failed fetch so a dead configs endpoint
  cannot repeatedly wedge request handling
Root cause of idle stalls: solveTraceless ran happy-dom + sync XHR (Atomics.wait) on the proxy main thread. After deep idle emptied the token pool, the next burst of requests spawned up to 8 concurrent solves x 6 retries, freezing the event loop for tens of seconds each time. Every connection hung until restart.

Now each solve runs in a dedicated worker:

- captcha-worker-entry.ts: worker entry running solveTraceless over postMessage (ESM)
- build-fork-worker.ts: pre-bundles the entry into a self-contained captcha-worker-entry.bundle.js (bun --target=bun --bundle), embedded into the compiled exe via with { type: file }
- captcha-solver.ts: worker-per-solve scheduler with 20s hard terminate, spawn/exit/error fallbacks

A hung solve can no longer wedge anything outside its own worker.
Verified: real request 200 with worker decode path, 3 concurrent requests all 200, error log clean; upstream tsc --noEmit clean, captcha unit tests 28/28.
State explicitly that this fork adds no telemetry, analytics, or
outbound reporting of any kind, and that debug/dump logs auto-redact
secrets (inherited from upstream).
The bundled worker (captcha-worker-entry.bundle.js, ~1.7MB / ~50k lines
minified-free) was committed as a build artifact. Upstream convention
keeps generated bundles out of git (.gitignore: Android server.cjs --
'CI/local builds regenerate it'), and PR CI only runs tsc + tests, so
the committed artifact added 50k lines of review noise without helping
any automated check.

- .gitignore: ignore src/proxy/captcha-worker-entry.bundle.js
- package.json: prebuild hook runs scripts/build-fork-worker.ts before
  every bun run build, so the artifact always exists at compile time
- captcha-solver.ts: comment points contributors to the regen path

Verified: fresh checkout without the bundle -> prebuild regenerates it
(1,683,850 bytes) -> tsc clean, full suite 885 pass / 0 fail.
@TriDefender
TriDefender merged commit 333dbcf into TriDefender:master Sep 27, 2026
TriDefender added a commit that referenced this pull request Sep 27, 2026
…ess fallback

The worker move (#55) embedded the entry via a static
`with { type: "file" }` asset import in captcha-solver.ts, which broke
every non-Bun-compile path:

- esbuild (Android server bundle) rejects the import attribute at parse
  time — build:android-bundle and the release build-android job failed
- a fresh checkout has no gitignored worker bundle, so dev mode / the
  Docker TS-source image failed to even LOAD captcha-solver on start-plan
- release.yml invokes `bun build --compile` directly (no npm lifecycle
  hooks), so the prebuild hook never ran and all five release binaries
  failed to resolve the asset

Restructure so the attribute syntax lives only in captcha-worker-asset.ts,
reached via dynamic import from captcha-worker-dispatch.ts with an
in-process solveTraceless fallback when the asset is unavailable (dev,
Docker, Android) or the worker entry fails to load (module-not-found
only; runtime crashes stay hard failures so an OOM never migrates to the
main thread). Mode transitions are announced once on stderr.

- package.json: every compile script self-generates the worker bundle;
  android esbuild marks the asset module --external (falls back on device)
- release.yml build-binaries generates the bundle before compiling
- Dockerfile bundles the entry at image build time
- tsconfig excludes the generated bundle from typechecking
- README privacy section reworded for upstream voice + README_EN parity

Verified: tsc clean; 888 tests (3 new dispatch tests, incl. a real
worker_threads round trip); android bundle builds; compiled exe embeds
the asset via the dynamic-import chain and solves through a worker
(6.7s live); no-bundle dev run degrades and solves in-process.

@TriDefender TriDefender left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified and merged, with a follow-up fix (88ac212) on master.

Verified locally on your branch (Bun 1.4.0, Windows x64):

  • tsc clean, 885/885 tests (CI-equivalent, no bundle present)
  • Real solve through a worker: 6.4s, 280-char param — the mechanism is solid

Gaps found and fixed in 88ac212 (none detectable by ci.yml, which is why they slipped through):

  1. esbuild rejects with { type: "file" } at parse time → build:android-bundle and the release build-android job failed even with the bundle present. Fix: the attribute now lives only in captcha-worker-asset.ts, dynamically imported; android esbuild marks it --external and the device falls back to in-process solving.
  2. Fresh checkout (no gitignored bundle) → captcha-solver failed to LOAD in dev mode and the Docker TS-source image on start-plan. Fix: dynamic asset resolution + in-process fallback (restores the lazy-load discipline too).
  3. release.yml invokes bun build --compile directly (no lifecycle hooks) → the prebuild hook never ran in the release pipeline and all 5 binaries failed to resolve the asset. Fix: generation chained into every compile script + an explicit release.yml step.
  4. Dockerfile now bundles the worker entry at image build time so containers get worker solving, not just the fallback.

Also reworded the README privacy section for upstream voice and added the README_EN counterpart (bilingual convention). Your timeout caps and the config-fetch negative cache are in as-is — good defensive work. The em-dash→-- comment churn was left alone.

Thanks for the excellent root-cause analysis in #54 — it made review straightforward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] start-plan: captcha pool refill freezes the main thread after idle; every connection hangs until restart

2 participants