Skip to content

Fix Unicode titlecase, folding and encoded case mappings - #38

Draft
284km wants to merge 1 commit into
mainfrom
fix/unicode-case-compatibility
Draft

284km wants to merge 1 commit into
mainfrom
fix/unicode-case-compatibility

Conversation

@284km

@284km 284km commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner

Unicode case conversion used uppercase-plus-lowercase as a substitute for titlecase, applied final-sigma context rules that Ruby does not use, and omitted identity exceptions from case folding. Correct the mappings for capitalize, swapcase and downcase(:fold), including Cherokee, titlecase digraphs and Turkic cases. Preserve the receiver encoding when ASCII input maps to non-ASCII bytes. Accept the valid single byte 0x80 in GBK/CP936 and CP949 when native operations validate their input.

Generate explicit titlecase and swapcase exceptions from the pinned reference Ruby, retain identity folding exceptions, and add a regression covering UTF-8/16/32, single-byte encodings, bang methods and multibyte validation.

Validation so far:

  • The complete upstream ruby/enc/test_case_comprehensive.rb file improved from 64 failures and 4 errors to 422 tests, 2,901,771 assertions, zero failures/errors/skips, within the unchanged 600-second limit (591.5 seconds including startup; 1,727,120 KiB peak process-family RSS).
  • The new regression matches Ruby 4.0.9 on macOS and Linux with hash indexing ON and OFF.
  • The final head passes all 523 corpus programs with hash indexing ON and OFF. An isolated final-head Unicode run also passes all 422 tests and 2,901,771 assertions in 590.6 seconds with 1,628,528 KiB peak process-family RSS. A concurrent-build run reached the 600-second wall limit, so the CI Unicode measurement still needs to confirm the available headroom. This PR remains draft pending final CI results.

@284km 284km left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed ad15079 against e45622a. No actionable findings introduced by this change.

The dedicated titlecase/swapcase exceptions and identity-preserving fold entries address the mapping differences directly. I also checked the case-method dispatch, bang-method behavior, legacy-encoding path, and every updated call to the double-byte validator.

Validation:

  • Fresh O2 build with the CI-pinned Mere 991f299b compiler and mgz revision.
  • Regenerated all five tables with Ruby 4.0.9: every table matches exactly, including 86 identity fold exceptions and 77 identity titlecase exceptions.
  • The new corpus regression matched Ruby with indexing ON/OFF, normally and with MERE_RUBY_GC_STRESS=100.
  • Exhaustive single-byte probes across the 18 affected encodings and all 256 byte values matched all 78,336 reference rows. These cover ordinary/:ascii/:turkic/:lithuanian options, folding, result bytes/encoding/validity, and bang results.
  • Mixed-string probes over the 401 distinct fold/title/swap exception codepoints matched all 20,050 mapping rows across UTF-8, UTF-16LE/BE, and UTF-32LE/BE, including ASCII/Turkic modes and folding.
  • The repository runner completed the focused upstream case-mapping/options files: 18/21 tests matched. The three residual failures also reproduce on the earlier baseline; the mapping file improved from 13/18 to 17/18. These residual compatibility failures are not attributed to this patch.
  • CI for this head reports 523/523 corpus programs matching Ruby and a successful Linux build. The comprehensive Unicode file is assigned to shard 5, which remains pending along with other jobs.
  • git diff --check passed.

I did not repeat the full comprehensive Unicode file or Linux execution locally. The reported full-suite result and its narrow margin under the 600-second limit still need the pending CI measurement to confirm the available headroom.

Comment thread tools/gen_case_tables.rb
when :swap then ch.downcase == ch ? ch.upcase : ch.downcase
else ch
end
next if out == fallback

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Positive: Comparing against each operation's fallback preserves identity exceptions that an out == ch filter would lose. Regeneration with Ruby 4.0.9 matches all five embedded tables exactly, including the Cherokee identity folds and titlecase exceptions.

Comment thread m_unicode.mere
let body =
(if enc_ascii_compat e && enc_multibyte e && (if enc_utf8_family e then false else true) then es_case_ascii (sget sh) e kind
else if sbcs_cased e && (if str_is_ascii sh then false else true) then sbcs_case (sget sh) e kind mode
else if sbcs_cased e && ((if str_is_ascii sh then false else true) || str_eq mode "t") then sbcs_case (sget sh) e kind mode

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Positive: An ASCII-only receiver still needs the single-byte encoder path when Turkic mapping can produce a non-ASCII byte. This condition preserves that encoding-specific behavior instead of copying UTF-8 output into a legacy-encoded result. The exhaustive byte/option probes matched Ruby across all 18 encodings handled by sbcs_cased, including bang methods.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant