Repository navigation
CI green: fix the root cause of each Linux test failure - #6
Merged
Merged
Conversation
MathF.Pow / Exp / Log / Log10 call the C runtime: UCRT on Windows, glibc on Linux. Their results differ in the last bit for roughly 0.1% of inputs (2M-sample probe: powf 1352-1771 diffs, logf 1884, expf 1391), which changed the gain map min/max fractions and a tone-mapped image hash on the Linux CI runner while Windows matched libavif. PortableMath evaluates them with IEEE double basic arithmetic only (2 atanh series log, Taylor exp after ln2 range reduction) and rounds to float once: bit-identical everywhere, and equal to the double-rounded value on every probed input (0 diffs in 4M on both OSes). AvifColorMath and the gain map compute / apply paths use it; every libavif golden is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Palette_RoundTrips_InOurDecoder pointed AV1_DUMPYUV at a hard-coded C:\Users\adamm\...\scratchpad path, so on any other machine the decoder's dump FileStream threw DirectoryNotFoundException. The dump now goes to a temp file that is read back and deleted, and the test asserts what its header always claimed: the decoded luma equals the source exactly. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The twin P/Invoked msvcrt.dll, which exists only on Windows (Linux CI: DllNotFoundException). There is no portable stand-in: the point is msvcrt's own unstable ordering. msvcrt's orderings for the fixed-seed inputs are now pinned as a digest, checked against the port everywhere; on Windows the live msvcrt.dll comparison still runs and also re-derives the digest, elsewhere it reports Skipped with the reason. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
It encodes a developer-supplied YUV (GQ_YUV, GQ_W, GQ_H, GQ_OUT...) and writes traces for an external libaom diff; without those variables it failed with ArgumentNullException from int.Parse on every machine. It asserts nothing by itself, so it is [Explicit] (runs only when selected) and names the missing variable when run without it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The twin sources lived only in a developer scratchpad, so CI had no aomtwin_inv / aomtwin_sp and the three tests failed on the unset SHARPIMAGE_AOMTWIN_INV / _SP. tests/native/aomtwin now holds twin_inv.c and twin_sp.c (portable exports and timer) and build.sh, which fetches libaom v3.14.1 (commit-checked), builds it static + PIC with libavif's local-libaom options and links the twins. CI caches the libaom build on build.sh and exports both paths. Running them on Linux exposed three test-side bugs, fixed here: - twin_fwd handed the caller's 8-byte aligned arrays and arbitrary stride (w + 8 / w + 16) to av1_lowbd_fwd_txfm, whose SIMD kernels use aligned loads/stores (the encoder's buffers are aom_memalign(32) with a block-width stride): SIGSEGV. It now repacks into a 32-byte aligned buffer with an aligned stride. - Every twin class installed its own DllImport resolver, and an assembly takes only one, so two configured twins in one run threw "A resolver is already set for the assembly". AomTwinNative is the single shared resolver. - OptimizeTxb_Captured passed without checking anything unless an 81 MB local capture was given. It now builds the records itself (random residual -> forward transform -> av1_quantize_fp, every tx size/type, both plane types, all contexts; all-zero blocks skipped as av1_optimize_b does) and still accepts a capture. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…unset mvhd/tkhd/mdhd times are 64-bit (version 1), but FromMp4Time only rejected values above long.MaxValue / 2, so anything past DateTimeOffset.MaxValue threw ArgumentOutOfRangeException from DateTimeOffset.FromUnixTimeSeconds instead of decoding. Found by DecoderFuzz on Linux (sharpimage_seq_12_422_alpha.avif); regression test patches a real file's time to 2^44 s. DecoderFuzz seeded each mutation with HashCode.Combine, whose seed is random per process, so the "fixed, seeded" quick budget was a different set of inputs every run and its failures could not be reproduced. It now mixes the seed deterministically. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…never pass Every twin class returned early, reporting a pass, when its library was not configured, and only inv / sp could be built outside a developer scratchpad. Now: - tests/native/aomtwin holds every twin source (aomtwin, cdef, hbd, ibc, lf, ml, pal + palsearch, pred, sf, inv, sp) with portable exports; build.sh builds them all against the pinned libaom (or an existing one via AOM_SRC / AOM_BUILD) on Linux and MSYS2 mingw-w64, and writes twins.env. CI exports it with SHARPIMAGE_AOMTWIN_REQUIRED=1, which turns a missing twin into a failure instead of a skip. - AomTwinNative: PathFromEnv, SkipUnless (Skipped with how to build the twins), SkipUnlessWindows; every twin loads through Register, including ibc and pal / palsearch (palsearch found with the platform extension). - aomtwin.c's quantize / forward / inverse / Hadamard / block error / SATD / sum-of-squares wrappers passed the managed (8-byte aligned, any stride) arrays straight to SIMD kernels that use aligned loads (libaom's encoder buffers are 32-byte aligned with block-width strides): SIGSEGV on Linux. They go through aligned copies now. - Windows-only, with the reason on other OSes (the reference only exists in the mingw-w64 build): ml NnSoftmax_Random, Expf_SoftmaxDomain, Log1pf_IntegersAndRandom, CnnPartition_HighBitdepthRandom (msvcrt exp and mingw x87 log1pf; glibc rounds differently) and pal MsvcrtQsort_ShortsortCallForCall (msvcrt qsort call for call; the digest test covers the port everywhere). - Inv / sp skip like the rest instead of failing; cdef still runs its AVX2-vs-C cross-check before skipping; bench-only and AVX2-only tests report Skipped with the reason. Linux (WSL, all twins): 114 twin tests, 106 passed, 8 skipped (6 above + 2 benches). Windows (mingw twins): 112 passed, 2 benches skipped. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- Developer harnesses that returned early unless a trigger file, corpus or env var was present are [Explicit] (run only when selected): Av1HbdVerify, Av1BitProfile, Av1CorpusTune, Av1EncodeBench, Av1SpeedExperiment, DecoderFuzz.ReplayOne / Campaign, JxlVarDct DecodeRealReferenceFile / RatioProbe / LargeImage / DeepTest, Jpeg_ByteExact_RealFiles. - Guards that returned when a committed test asset was missing are gone (peppers_rgba.png, photo_small.*, landscape.jpg, bounce_anim.gif are all in TestAssets): a missing asset now fails. - AVX2 / SSSE3 comparisons report Skipped on CPUs without the ISA. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…oat) PortableMathD gives pow / exp / log / log2 / log10 / cbrt for doubles from IEEE basic arithmetic and FMA only: double-double, table-driven (128-entry log, 64-entry 2^(j/64)), so results are bit-identical on every OS and equal the correctly rounded double on every probed input (0 misses in 300k per function against a slow double-double series reference; UCRT misses 13 to 1473 and glibc 7 to 2304 of the same). Fast variants (LogFast, Log2Fast, Log10Fast, ExpFast, PowFast) trade the last bit for speed and stay platform-independent, for heuristics. PortableMath (float) is rewritten on 64-entry log2 / exp2 tables and short polynomials (relative error ~2^-52): 0 differences from the correctly rounded float in 4M samples per function, about MathF's speed, cbrt via a Halley step (2.5x faster than MathF.Cbrt). Adds Cbrt, LogFast, Log2Fast. The tables are literal data (no static initialisation, so hot callers carry no class-init checks); Compute*TableBits rebuild them from their definitions and the tests check they match. The kernels are AggressiveOptimization so short-lived callers never run them at tier 0 (4x slower there). QuantumLut: per-quantum-value tables built in parallel, for per-sample functions of the sample alone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…hs fast Every MathF / Math Pow, Exp, Log, Log2, Log10 and Cbrt in src goes through the portable kernels, so results no longer depend on UCRT vs glibc: colour conversions, effects, AV1 encoder (TPL, temporal filter, rate control, ML expf / log1pf), JXL encoder, HEVC lambda, gain maps. Exceptions: Av1StillImageEncoder.cs / .LumaSearch.cs / .Palette.cs (being edited elsewhere) and trigonometric calls. Powers of two in HdrCoder and JxlBitReader become exact MathF.ScaleB. Every existing golden still passes on Windows and Linux. To avoid regressions where the call sits in a per-sample loop: - ColorAdjust.ApplyPerPixel (Gamma, Levels, ...), MathOps.Evaluate's transcendental operators, CDL, PerceptualDiff's linearisation and SrgbConverter.Decode use per-quantum-value tables (QuantumLut), and AutoGamma / Levelize build theirs in parallel: same expressions, so the same results. - ToneMapReinhard sums its log-luminance per row in parallel and adds the rows in order (deterministic). - SmartCrop precomputes p log2 p for full windows. - Heuristic estimates (JXL entropy costs, SmartCrop, Drago, Liquify, noise) use the fast variants. Release, 1024x768 16-bit RGB, best of 3 rounds (ms, before -> after): Gamma 4.21 -> 1.54, Levels 4.41 -> 1.46, AutoGamma 4.54 -> 3.70, Levelize 1.33 -> 0.98, EvaluatePow 3.43 -> 1.16, ApplyCdl 3.94 -> 3.93, Reinhard 6.50 -> 4.63, Drago 4.41 -> 4.07, PerceptualDiff 270 -> 205, SmartCrop ~40 -> ~36, Oklab 1M 31.1 -> 28.4, SrgbConverter.Decode 1M 10.9 -> 1.0, AVIF gain map 164 -> 76, AVIF sequence 519 -> 477, JXL lossy 3577 -> 3546. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Echostorm44
marked this pull request as ready for review
October 7, 2026 23:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes CI green by fixing the root cause of each of the 11 failures of run 37676215845 (ubuntu). No blanket skips.
ToneMapped_MatchesLibavifExactly(1 case),EncodedGainMap_MatchesLibavifCombine(4 cases)MathF.Pow/Exp/Logcall the C runtime; UCRT and glibc differ in the last bit for ~0.1% of inputs (2M-sample probe), which shifted gain-map fractions and a tone-mapped hash on LinuxCore/PortableMath: float pow/exp/log/log2/log10 from IEEE double basic ops only, rounded once; bit-identical on both OSes and equal to the double-rounded result on every probed input. Used byAvifColorMathand the gain-map paths. All goldens unchanged.MsvcrtQsort_MatchesMsvcrtDllmsvcrt.dll(Windows only)EncodesSequence(also failed on Windows)GQ_*env vars[Explicit]+ names the missing variableInvTxfm_AllSizesTypesEobs_MatchLibaom,FwdTxfm_AllSizesTypes_MatchLibaom,OptimizeTxb_Captured_MatchLibaomtests/native/aomtwin(sources +build.sh: pinned libaom v3.14.1, commit-checked); CI builds them (libaom cached) and sets the env vars. Fixed on the way: twin_fwd misaligned SIMD input (SIGSEGV on Linux), one-resolver-per-assembly clash between twin classes, OptimizeTxb now generates its own blocks instead of passing silently without an 81 MB local capturePalette_RoundTrips_InOurDecoderC:\Users\...\scratchpaddump pathAlso found while running the full suite on Linux:
DecoderFuzzseeded withHashCode.Combine(random per process, so the "fixed" budget changed every run) and it hit a real bug: a 64-bit mvhd time past year 9999 threwArgumentOutOfRangeException. Both fixed, with a regression test.🤖 Generated with Claude Code