recipes: curl-cffi 0.16.2 + flet-libcurl-impersonate 2.1.1 - #117
Merged
Conversation
unpack_source already honors source.strip (the tar --strip-components equivalent), but the schema locked the url-source object to url-only, so no recipe could set it. Expose it: strip: 0 unpacks a prebuilt release tarball whose files sit at the archive root verbatim, instead of silently dropping the root-level files under the default strip=1 (which assumes a single top-level wrapper directory).
Adds TLS-impersonation HTTP (curl_cffi, flet#6768) for iOS and Android. flet-libcurl-impersonate repackages lexiforest/curl-impersonate's prebuilt static mega-archive (patched curl 8.21.0 + BoringSSL + nghttp2/nghttp3 + ngtcp2 QUIC + brotli + zstd + zlib, all in one libcurl-impersonate.a) per slice; source.strip: 0 because the tarballs carry the .a + include/curl/ at the archive root. armeabi-v7a is excluded (no upstream 32-bit ARM binary). curl-cffi links that archive into its cffi _wrapper extension. mobile.patch adds one opt-in lever, IMPERSONATE_FORGE_TARGET, so scripts/build.py selects the target arch + static-link recipe from the forge target instead of the build host's uname (which is never the target under cross-compilation), and takes the prebuilt .a from the flet-libcurl-impersonate host_build dep via IMPERSONATE_BUILD_DIR (no network in the build). On iOS -- the first-ever curl-cffi iOS build -- it also links the Apple system libraries the archive needs (Security + CoreFoundation for USE_APPLE_SECTRUST, libiconv + libicucore for USE_APPLE_IDN). Built and verified on-device for all five slices (android arm64-v8a/x86_64, iOS device arm64, iOS sim arm64/x86_64): import loads the extension, the linked libcurl reports the impersonate build, and curl_easy_impersonate() applies a browser fingerprint. Runtime deps cffi>=2.0.0 + certifi resolve from pypi.flet.dev. [skip ci]
…ple-frameworks, source.strip:0) Fold the reusable findings from the curl-cffi recipe into the skills: - new-mobile-recipe: a shape row + deep-dive for a cffi/ctypes consumer whose own build self-detects arch via the host uname and downloads a prebuilt static lib (steer it off the forge target; stage the lib via a source.strip:0 flet-lib* dep). - forge-error-catalogue: iOS undefined _SecTrust*/_iconv/_uidna_* when linking an Apple-built static archive -> link Security/CoreFoundation/iconv/icucore; and the source.url root-level-tarball -> source.strip:0 fix. [skip ci]
`flet-libcurl-impersonate` stopped building when #116 turned the missing-licence warning into an error: upstream's release tarballs are a bare `libcurl-impersonate.a` plus headers, with no notice anywhere in the archive, the release or CMakeLists.txt. `about.license_file: []` is the one option not available here — the wheel really does carry the object code, and curl, BoringSSL, brotli and zstd all attach notice conditions to binary redistribution. So the notices are vendored, one file per component, fetched at each component's pinned version tag. The component list is upstream's `CMakeLists.txt`, confirmed against the archive itself: `ar t` is no help on a single `ld -r` blob, but the merge left 665 `STT_FILE` symbols and the build's own `deps/src/<project>/` paths in `.rodata`, which attribute every object. Looked for and absent: c-ares, libpsl, libidn2, libssh2, wolfSSL, mbedTLS, GnuTLS, rustls — curl's own backend files for those are present but compile to empty translation units. BoringSSL is the load-bearing one. It is Apache-2.0 at this commit, not ISC and not the OpenSSL licence (the `SSLeay` strings in the binary are API-compat shims), so §4(d)'s NOTICE requirement and its patent grant are the only non-inert terms in the set. zstd's GPL-2.0 alternative is deliberately not in the expression: the dual licence is a fact about the project, not about this artifact, and we take the BSD arm. Two file-choice traps, both the libiconv trap in different directions — nghttp2's `LICENSE` is a 12-byte "See COPYING", and zstd's `COPYING` is 18 KB of GPL-2.0.
The library version tracks curl-cffi's own pin rather than the newest release: `scripts/build.py::__version__` is "2.1.1" at 0.16.2, and that one line is the only change to the file the patch touches. Taking curl-impersonate 2.2.0 instead would have been silent — our patch skips the download that constant normally drives, and `ffi/cdef.c` compiles against curl-cffi's own bundled `include/curl/`, so a skew links a mismatched archive and still goes green. meta.yaml now says so next to the pin. `build.number` 0 -> 1 on both. These were the only two `number: 0` recipes in the tree against a schema default of 1, and 0 is not a smaller build number — for the Python path `build.py` only passes `--build-number` when the value is truthy, so curl-cffi's wheel shipped with no build tag at all and lost the PEP 427 tie-break. That matters here specifically: PyPI carries `curl_cffi-0.16.2-cp313/cp314-\ android_24_arm64_v8a`, so `flet build` sees two builds of one version. CI could not have caught it either — the mobile test rewrites local wheels' build tag to 9999. Adds `test_default_cacert_resolves_to_a_readable_file`, which backs the README's certificate claims on device: curl-cffi resolves its CA bundle once at import and hands the path to libcurl, so if that path were not a readable PEM every HTTPS request would fail with CURLE_SSL_CACERT_BADFILE instead of anything diagnosable. Verified on device, both platforms, with the recipe-tester and a throwaway probe: 4/4 tests EXIT 0 on an Android emulator (arm64, API 34) and an iOS simulator, and a real `curl_cffi.get(..., impersonate="chrome")` returns 200 on both. No certifi wiring is needed — `flet build`'s generated `lib/python.dart` exports SSL_CERT_FILE before the app runs, and certifi's zipimport path extracts `cacert.pem` to a real file anyway.
The recipe README per README.rst § Documenting a recipe, and a runnable example that fans five browser profiles at one fingerprinting endpoint under a single `asyncio.gather` — the shape an app querying several endpoints at once actually needs, and the one thing no desktop run can establish, since JA3/JA4 hash the ClientHello and only the far end of a handshake can report it. Measured on an Android emulator rather than asserted: 5/5 probes, 886 ms wall against 3,208 ms summed, five distinct JA4/JA3n/JA3 triples, `off` sending no User-Agent at all, and `chrome150`/`chrome131_android`/`off` sharing one Akamai HTTP/2 hash — two clients can share an h2 fingerprint and be nothing alike on the wire. The page carries the mandatory `target_arch` note (no armeabi-v7a wheel exists, and `flet build apk` defaults to all three ABIs), and a Licensing bullet, because the one thing a reader cannot discover is that an MIT package's extension statically contains nine projects whose notices ship in a different wheel. The certificates section is the part worth reading twice, and it is the opposite of what it looks like: nothing needs configuring, because Flet sets SSL_CERT_FILE before the app runs. What does bite is that the device then trusts certifi's roots and only those, so a corporate root or mitmproxy that works under `flet run` fails on a phone.
The catalogue covered upstream renaming or relocating its notice, and the `[]` opt-out. It did not cover a prebuilt-repackage recipe whose upstream ships no notice anywhere, where `[]` is exactly wrong and the answer is to vendor the notices into the recipe dir. Adds that, the symbol-probe recipe for recovering a mega-archive's component list, and the two file-choice traps (a stub `LICENSE` redirect, and a dual-licensed project's `COPYING` being the arm we do not take).
…kip ci] `requires-python = ">=3.10"` was the wrong default here. `flet build` takes the HIGHEST stable Python the spec admits (`python_versions.py`: `max(stable_matching)`), so it landed on 3.14 — precisely the range where PyPI also carries a curl-cffi `android_24_arm64_v8a` wheel at the same version, and the example that exists to demonstrate this recipe's wheel might not have contained it on the flagship ABI. Upstream ships no cp312 mobile wheel at all, so `==3.12.*` is the lever that settles it. The README said to pin `curl-cffi` instead, two sentences after explaining that a version pin cannot separate two builds of one version; it now names the Python pin. A failed probe rendered 153 characters, 74 of them libcurl's docs URL, on five stacked rows. Trimmed at the URL and capped at two lines: 76 characters, still naming the host and the curl error number. `certifi` is imported directly for the CA-bundle readout, so it is declared rather than relied on transitively through curl-cffi. Docstrings run through docformatter to match the other examples, and the test's `import curl_cffi` is now asserted on rather than tripping F401 — importing it is the point of that test.
…tcha [skip ci] recipe-patterns already said not to write `number: 0`. It did not say that the lost build tag forfeits a PEP 427 tie-break against upstream's own mobile wheels on an extra index, or — the part that makes it expensive — that CI cannot catch it, because the mobile test rewrites local build tags to 9999 before installing. Only the filename in dist/ tells you. Also records that a version bump resets the number to 1. Corrects the curl-cffi shape's on-device count, and notes that a real impersonated HTTPS request was verified on both platforms — that one is worth having written down, because the intuition that Flet 0.86's sitepackages.zip breaks certifi is wrong and cost this recipe a wrong README draft before a device run settled it.
Run 33395094861: 10/10 jobs, 0.16.2 across 3.12/3.13/3.14 on all five slices, on-device 4/4 EXIT 0 on both the CI emulator and simulator.
The warning that `about` is ignored on non-build.sh recipes fired on almost every
Python recipe in the tree — gdal, pyproj, pyogrio, rasterio and the rest — none of
which declare it. Schema validation fills in defaults in place, and `about`'s children
default to empty strings rather than to nothing, so by the time `fix_wheel` looked,
every recipe had `about = {"license_file": "", "license": ""}` and the truthiness test
could never be False.
Net effect was that a warning written to catch one specific mistake became noise on
every build, which is the reliable way to make the real case invisible.
Records whether the recipe declared the key before defaults are merged, and tests
that. Testing the values instead would have been close but not right: it would miss
`about.license_file: []`, the deliberate opt-out, which is a thing an author wrote and
which does nothing on this path.
Verified by building lru-dict (a Python recipe) both ways — no warning as-is, warning
present with an `about:` block appended — and flet-libomp, where the build.sh path
still emits `License-Expression: Apache-2.0 WITH LLVM-exception` and its LICENSE file.
…ices into one Nine notices sitting in the recipe root drowned out meta.yaml and build.sh, and the meta.yaml list naming them was a second copy of the folder's contents that a bump could put out of step. A folder settles both, alongside tests/ and examples/: everything in recipes/<name>/licenses/ ships, whatever each file is named, so the folder IS the list. The names are deliberately not licence-shaped — COPYING.curl, LICENSE.boringssl — because a recipe supplying notices for a prebuilt binary chooses them per bundled project, so the LICEN[CS]E/COPYING/COPYRIGHT/NOTICE match that guards top-level discovery would be the wrong test inside a folder that exists only to hold notices. The folder name is stripped from the destination, so a notice lands at dist-info/licenses/<name> rather than the doubled dist-info/licenses/licenses/<name> that preserving the relative path would give — which is what pushed these files into the recipe root in the first place. An explicit about.license_file naming a licenses/ path is stripped the same way, so both routes agree; a path into the SOURCE tree keeps its relative path, as flet-libjq's modules/oniguruma/COPYING needs. Verified: flet-libcurl-impersonate ships all nine at dist-info/licenses/<name> with nine License-File headers and no doubled paths, and flet-libomp's top-level recipe LICENSE still comes through. Every branch exercised against scratch trees — folder alone, folder plus an upstream LICENSE, a nested file, upstream-only, nothing at all (raises), explicit naming a licenses/ path, explicit source subpath, the [] opt-out, and an explicit missing file.
…[skip ci] Session, FileCacheBackend, Curl, ImpersonateError and CertificateVerifyError were named without links, against README.rst's rule to link every API reference the first time it appears. All five resolve to upstream's api.html anchors; `requests` now points at curl-cffi's own vs-requests page and certifi at its repository. FingerprintManager and config_warnings stay unlinked deliberately — upstream documents neither, and a guessed anchor is worse than none. The Build notes opener said patches/mobile.patch explains its own hunks and meta.yaml justifies its settings inline. Both are true, both are this repo's standing convention, and the reader of a maintainer section has both files open — so the paragraph spent three lines telling them nothing. The section now starts at Recipe shape. All 31 links and all 16 anchors verified to resolve.
…o 3.12 [skip ci]
The page said an unpinned resolve's preference "was not measured" and that a `==` pin
could not separate two builds of one version, then told the reader to pin Python instead.
Measured now, with `pip download --platform android_24_arm64_v8a` against both indexes:
pyzmq==27.1.0 cp313, identical platform tags, only a build tag between them
-> pyzmq-27.1.0-1-cp313-cp313-android_24_arm64_v8a.whl (this index)
lru-dict==1.4.1 cp313, android_24 here vs android_21 upstream
-> lru_dict-1.4.1-1-cp313-cp313-android_24_arm64_v8a.whl (this index)
pyzmq unpinned, upstream carries 27.2.0 and this index 27.1.0
-> pyzmq-27.2.0-cp313-cp313-android_24_arm64_v8a.whl (upstream)
So a forge wheel is not the lower-priority candidate: at equal version it wins on both
the build tag and the higher android_24 platform tag. Version is compared first, so the
only thing that flips it is upstream shipping a version this index has not caught up to
— and because upstream ships arm64-v8a alone, that yields a MIXED build rather than
theirs outright. A `==` pin is therefore exactly the right lever, which is what the
example already declares.
`requires-python` goes back to `>=3.12`. Forcing 3.12 to dodge upstream's cp313/cp314
wheels was solving a problem that does not exist, at the cost of denying the example the
newer runtimes this index builds for.
…[skip ci] Version is compared before any tag, so an upstream release the recipe has not caught up to wins outright — and because upstream ships arm64-v8a alone, that produces a mixed app rather than theirs outright. Measured against both indexes with pip download.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds recipes for curl-cffi 0.16.2 — an HTTP client that binds through cffi to curl-impersonate, a patched libcurl built against BoringSSL, so the TLS ClientHello, the HTTP/2 settings and the header order that go out on the wire are a named browser's rather than a Python library's. Requested in flet#6768 and again in flet#6804.
These are the first iOS curl-cffi wheels anywhere — upstream has never shipped iOS, and publishes only a single Android ABI. Reach for it in a Flet app when an API or page answers an ordinary client with a challenge page, a 403 or a silent block.
Recipe shape
Two recipes, because the payload is a prebuilt binary rather than a build.
flet-libcurl-impersonaterepackages upstream's per-slice release tarball — a singlelibcurl-impersonate.amerging nine projects — into{platlib}/opt, and curl-cffi links it throughrequirements.host_build, so it never becomes aRequires-Distand ships nothing to the device.unzip -lon the consumer wheel shows 36 files and noopt/, against 25 files and a 43 MB archive in the library wheel it linked against.Building curl and BoringSSL from source across five slices was rejected: BoringSSL's build system is a project of its own, and the result would drift from the fingerprints upstream actually tests. The cost is that upstream's release assets are the supply chain, which is what
### Upgrade hazardsin the README is mostly about.excluded_arches: [armeabi-v7a]on both — upstream ships no 32-bit ARM build of curl-impersonate — which is why the README makestarget_archmandatory rather than advisory.Three forge changes
source.stripexposed in the meta-schema.build.pyalready read it; the object-sourceadditionalProperties: falselocked it out. These release tarballs put files at the archive root, so the defaultstrip: 1silently dropped the.aand kept only the headers.A recipe's
licenses/folder is now collected. Everything inrecipes/<name>/licenses/ships, under any name, with the folder name stripped from the destination so notices land atdist-info/licenses/<name>rather than doubled. The names are deliberately not licence-shaped (COPYING.curl,LICENSE.boringssl), so theLICEN[CS]E|COPYING|COPYRIGHT|NOTICEmatch that guards top-level discovery would be the wrong test inside a folder that exists only to hold notices. Purely additive: onlyflet-libomphas a recipe-root licence file, and all five recipes settingabout.license_filename paths in the source tree.The
aboutwarning no longer fires on every Python recipe. Schema validation fills in defaults in place, andabout's children default to empty strings, soif (meta.get("about") or {})could never be False —gdal,pyproj,pyogrioandrasterioall emitted it despite declaring nothing. It now records whether the recipe declared the key before defaults are merged; testing the values instead would misslicense_file: [], the deliberate opt-out.The library version tracks curl-cffi's own pin
scripts/build.py::__version__is"2.1.1"at curl-cffi 0.16.2, and that one line is the only change to the file our patch touches between 0.16.0 and 0.16.2. curl-impersonate 2.2.0 exists, and taking it would fail silently: the patch skips the download that constant normally drives, andffi/cdef.ccompiles against curl-cffi's own bundledinclude/curl/(which also changed in 0.16.2), so a skew links a mismatched archive and still produces a green wheel.meta.yamlrecords the lockstep rule next to the pin.Licences
flet-libcurl-impersonatestopped building under #116: the release tarballs are a bare.aplus headers, with no notice in the archive, the release, orCMakeLists.txt.about.license_file: []is the one option not available here — the wheel really does carry the object code, and curl, BoringSSL, brotli and zstd all attach notice conditions to binary redistribution.So nine notices are vendored, one file per component, fetched at each component's pinned version tag. The component list is upstream's
CMakeLists.txt, confirmed against the archive itself —ar tis no help on a singleld -rblob, but the merge left 665STT_FILEsymbols and the build's owndeps/src/<project>/paths in.rodata, which attribute every object. Looked for and absent: c-ares, libpsl, libidn2, libssh2, wolfSSL, mbedTLS, GnuTLS, rustls — curl's own backend files for those are present but compile to empty translation units.BoringSSL is the load-bearing one. It is Apache-2.0 at this commit, not ISC and not the OpenSSL licence (the
SSLeaystrings in the binary are API-compat shims), so §4(d)'s NOTICE requirement and its patent grant are the only non-inert terms in the set. zstd's GPL-2.0 alternative is deliberately not in the expression: the dual licence is a fact about the project, not about this artifact, and we take the BSD arm.Two file-choice traps, both the libiconv trap in different directions — nghttp2's
LICENSEis a 12-byte "See COPYING", and zstd'sCOPYINGis 18 KB of GPL-2.0.Validation
CI 10/10 across 3.12/3.13/3.14 on all five slices, dispatched as a chain (
packages=curl-cffi:,prebuild_recipes=flet-libcurl-impersonate) since the library is abuild.shrecipe and the canonical-python gate would otherwise strand the 3.13/3.14 legs. On-device 4/4 EXIT 0 on both platforms, on CI's emulator and simulator and again locally (Android arm64 API 34, iOS 18.6 simulator).Wheels are 2.6–2.9 MB compressed / 6.3–7.0 MB unpacked per slice. Android
DT_NEEDEDislibm,libpython3.<minor>,libc++_shared,libdl,libc; the iOS extension links system frameworks only (Security, CoreFoundation, libiconv, libicucore, libc++, libSystem),platform 2, no stray dylibs.Beyond the test suite, the example ran a real fan-out on device: 5/5 probes at 200 over h2, 886 ms wall against 3,208 ms summed, five distinct JA4/JA3n/JA3 triples, and
offsending noUser-Agentat all.Real impersonated HTTPS works on both platforms with no app configuration.
flet build's generatedlib/python.dartexportsSSL_CERT_FILEbefore the app runs, and certifi'simportlib.resourcespath extractscacert.pemto a real file even from inside Android'ssitepackages.zip— so there is noextract_packagesentry to write.test_default_cacert_resolves_to_a_readable_filekeeps that honest on device rather than leaving it as prose.Upstream also publishes an Android
arm64-v8awheel oncp313/cp314. Measured withpip download --platform android_24_arm64_v8aagainst both indexes: at equal version this index wins on both the build tag and the higherandroid_24platform tag. Version is compared first, so the only thing that flips it is upstream releasing a version this index has not caught up to — and since upstream ships one ABI, that yields a mixed app rather than theirs outright. The README says so and the example pins accordingly.