Repository navigation
feat(captcha): pluggable solver backends (happy-dom happy path + native reference) - #29
Conversation
…wright, native reference Adds captcha_node/ — a standalone Node.js solver runtime driven by a daemon (bounded-concurrency worker pool), plus a backend switch in src/proxy/captcha.ts via ZCODE_CAPTCHA_BACKEND: - happy (happy-dom): production-proven happy path. ~815ms CPU/solve, ~260-330ms with CAPTCHA_WINDOW_REUSE=1. No browser install, more IP-tolerant than jsdom. Validated end-to-end live (verify tokens minted). - jsdom: unchanged default — the existing in-process path in captcha.ts still handles ZCODE_CAPTCHA_BACKEND=jsdom (or unset). - playwright: Chromium backend for last-resort compatibility debugging. - native (native_solve2.js): pure-HTTP solver (~600ms, no DOM). Currently rejected upstream with F001 from most IPs — included as a working protocol reference; not wired into the default flow. The daemon keeps DOM/Chromium work out of the Bun event loop and bounds memory by recycling workers (RSS cap + max-solves per worker, duplicate certifyId rejection). Full backend details and benchmarks: captcha_node/README.md
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 990ab9dcd3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // idle reaps (entry.reaping, idle system) do not respawn. | ||
| if (USE_POOL && !entry.reaping) { | ||
| if (queue.length > 0 || active > 0 || workers.length < WORKER_WARM_FLOOR) { | ||
| workers.push(spawnWorker()); |
There was a problem hiding this comment.
When any pooled worker exits without entry.reaping (the default recycle path after ZCODE_CAPTCHA_MAX_SOLVES/RSS and crashes), this calls workers.push(spawnWorker()), but spawnWorker() already appends the new entry to workers before returning. Each respawn therefore leaves duplicate/stale entries; after several recycles workers.length can reach MAX_CONCURRENT even though no writable worker exists, so drainQueue() stops spawning replacements and queued captcha solves hang until the caller times out.
Useful? React with 👍 / 👎.
| // ceiling — either bound keeps memory strictly bounded per worker. | ||
| solvesCount++; | ||
| if (solvesCount >= MAX_SOLVES_PER_WORKER || (WORKER_MAX_RSS_MB > 0 && workerRssMb() > WORKER_MAX_RSS_MB)) { | ||
| setTimeout(() => process.exit(0), 10); |
There was a problem hiding this comment.
When a worker crosses the max-solve/RSS threshold under sustained queue pressure, it emits a successful result and only schedules process.exit 10 ms later. The daemon treats that result as making the worker idle and immediately calls drainQueue(), so it can assign the next request to this same process just before it exits, returning worker died mid-solve for a token that should have been handled by a fresh worker.
Useful? React with 👍 / 👎.
| import { fileURLToPath } from "node:url"; | ||
|
|
||
| const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../.."); | ||
| const CAPTCHA_NODE_DIR = path.join(ROOT, "captcha_node"); |
There was a problem hiding this comment.
In the published Docker flow, the image installs only the root package and copies only src/ plus root config files (Dockerfile lines 5-12); it never copies captcha_node/ or installs this nested package. With ZCODE_CAPTCHA_BACKEND=happy or playwright, this new path resolves to /app/captcha_node, so runCaptchaSolve() fails with missing daemon.js/solver.js (and would lack happy-dom/playwright even if copied) for all captcha requests in that container.
Useful? React with 👍 / 👎.
|
LGTM👍 |
|
the happy dom is actually really good for this takes way less memory and cpu compared to jsdom |
Adds a CaptchaTokenPool that pre-solves tokens in the background and serves them in ~8-12ms on the request hot path, instead of paying a full ~1.5-10s solve per request. For daemon backends (ZCODE_CAPTCHA_BACKEND=happy|playwright), getCaptchaToken() now takes from the pool; the jsdom default keeps the existing single-token cache. Pool behavior (src/proxy/captcha-jsdom.ts): - demand-driven sizing: target grows with take rate (up to poolSizeMax), decays after idle, deep-idles to zero traffic after 15 min - empty-pool takes race parallel solves (cap via CAPTCHA_SOLVE_RACE_DEADLINE_MS, default 25s) - per-token TTL with expiry pruning; duplicate certifyId rejection (Aliyun F008 guard) across pool and issued tokens - CPU governor bounds solve concurrency to the host CPU budget (CAPTCHA_CPU_LIMIT_PCT, default 100) - IP-block detection hook (onCaptchaIpBlock) for the Aliyun 'too many captcha requests' family Proxy startup (start-plan) warms the pool in the background (CAPTCHA_POOL_MIN, default 20). env-only configuration; no config.yaml schema changes in this PR. Verified live: prefill 2 -> two takes in 8ms/12ms with distinct certifyIds and automatic background refill; 500 bun tests pass (23 new); upstream compile build OK.
|
Added a second commit with the request-scaling pre-solved token pool — the methodology that keeps captcha solving off the request hot path:
|
|
Alright before anything else I need this to be selfcontained, the release workflow builds and releases self contained executables. This means no external dependencies, in this PR, Node.js is required on the host and someone downloading a self-contained binary to avoid installing runtimes doesn't have it. For the moment this would only ru on the Android-APP since it's a node app and ships the node runtime, but canvas/playwright/Chromium would never work there. |
|
makes sense, i didnt mean this to be a ready to merge pr but an idea of what was possible for efficiency and what i've found from my multiple days of ai work. I'll make this self contained and commit. |
Ironic, since Android is the one target that does bundle a Node binary (libnode.so), but nothing sets ZCODE_NODE_PATH to it |
|
Resolve the containment issue and it's good to merge. |
|
i get a very low (lower than zcode itself) ttfb with this scaling and solver backends while staying under 5% cpu usage while zcode kills my pc |
Open upstream PR only. Head b50b45a.
… external Node The happy-dom solver now runs INSIDE the Bun process (src/proxy/ captcha-happy.ts), so the release binaries stay self-contained: no external Node.js, no captcha_node/ daemon, no canvas/ playwright/Chromium. happy is the default backend. Bun-compat work in the port: - GlobalWindow instead of VM Window (Bun drops happy-dom's VMGlobalPropertyScript assignments; class-field globals work) - globalThis window aliasing (ref-counted across concurrent solves) so guest scripts can resolve bare window/document/XMLHttpRequest/ Range/... identifiers — under Bun, script tags execute in the host realm; host-critical globals (process, Bun, intrinsics, timers registry is window-bound) are never shadowed - guest timers bound to the window's own registry so pe-VM callbacks die with the window instead of firing after destroyDom - sync XHR served by a SharedArrayBuffer + Atomics blocking fetch on a worker thread — happy-dom's built-in sync fetch spawns 'process.argv[0] -e', which breaks compiled binaries - Option/Video element-constructor polyfills (FeiLin references them) - uncaughtException handler: a bad rotated pe bundle fails one solve, never the proxy process Verified: bun test 500/0; in-process solves 6/6 one-shot (Node solver parity in the same window); compiled binary (no Node on PATH, fresh HOME, cold CDN cache) boots start-plan, warms the pool, and mints real T001-verified tokens; pool prefill+take e2e ~0-1ms hot path.
|
Containment resolved — the solver is now fully self-contained. New commit rewrites the happy-dom backend to run in-process inside the Bun binary:
Verification (all from this exact tree):
|
| console.log(`zcode-proxy listening on ${url}`); | ||
| if (config.plan === "start-plan") { | ||
| // Pre-solve the captcha token pool in the background so first requests | ||
| // don't pay the full solve latency. No-op unless a daemon backend |
There was a problem hiding this comment.
Stale comment still says "No-op unless a daemon backend (ZCODE_CAPTCHA_BACKEND=happy|playwright) is configured"; playwright no longer exists and happy is the default.
|
I will remove the jsdom seperately since it's deprecated and we have happy working. This PR is good to merge, stale comments will be removed later. |
3 participants
What
Adds a standalone Node.js captcha solver runtime (
captcha_node/) and a backend switch insrc/proxy/captcha.ts, selected viaZCODE_CAPTCHA_BACKEND:happyjsdom/ unsetcaptcha.tsstill handles itplaywrightnative(native_solve2.js)happyruns in a dedicated Node daemon (daemon.js→worker.js) with a bounded-concurrency worker pool, worker recycling (RSS cap + max solves per worker), duplicate-certifyId rejection, and optional window reuse (CAPTCHA_WINDOW_REUSE=1→ ~260-330ms CPU/solve). This keeps DOM work out of the Bun event loop so request serving never waits on or competes with a solve.Benchmarks (measured 2026-08-22)
captcha_node/README.mddocuments the mutation-test findings on why pure-HTTP minting is blocked by designScope
Captcha backend only — no other proxy features are included. Default behavior is unchanged (
ZCODE_CAPTCHA_BACKENDunset → existing jsdom path).Verification
bun test: 477 pass / 0 fail (upstream suite)bun build --compile(upstream build script): successcertifyId+securityTokentokens via both the one-shot CLI and the daemon JSON-line protocolcd captcha_node && npm install && ZCODE_CAPTCHA_BACKEND=happy node solver.js 11xygtvd sgp no8xfe