.notdef must not claim to be a character - #7
Merged
Conversation
A character the font does not have is written as glyph 0, .notdef. The first such character also took glyph 0's /ToUnicode entry, and every later missing character then read back as that first one — so the text a foreign reader extracted was not text with holes in it but text that was confidently wrong, which is the worse of the two failures. Found by writing the lines a text writer gets wrong and letting poppler say what it could read back. Written here, read by pdftotext: a 漢字 b かな c 한글 d -> a 漢漢 b 漢漢 c 漢漢 d السلام عليكم -> one Arabic letter, eleven times a U+1F600 b U+1D400 c -> a U+1F600 b U+1F600 c Every missing character in a document became whichever missing character came first in it. WHY THE REPLACEMENT CHARACTER RATHER THAN NOTHING Two honest answers were available and both were measured rather than argued about. Omitting the entry gives: a 漢字 b かな c 한글 d -> abcd السلام عليكم -> (nothing at all) which loses the spaces along with the characters and leaves a reader no way to know anything had been there. Mapping glyph 0 to U+FFFD gives: a 漢字 b かな c 한글 d -> a __ b __ c __ d (U+FFFD each) السلام عليكم -> eleven of them The deciding measurement is what the page itself shows. Rendered at 72 dots to the inch, the page draws 17 856 dark pixels either way: the face's own .notdef draws a box, so the loss is already visible to a person looking at it. Text read off that page should say what the page says — one mark per missing character, in the right place — and U+FFFD is how Unicode says exactly that. Nothing would have made the extraction disagree with the picture. WHAT IS NOT AFFECTED Both call sites are covered: Text and TextShaped mark through the same fontUse. Everything the font does have is unchanged — Cyrillic, Greek, the punctuation a producer substitutes, combining accents, astral-plane characters that the font carries, and two thousand characters on one line all still read back exactly. Ten files across five faces, plain and shaped, are parsed and drawn by poppler with the text byte-identical to what was written, and pdffonts reports every one as embedded, subsetted and with a Unicode map. Two existing tests asserted the old contract and now assert the new one: .notdef is always mapped, so 150 mapped glyphs make blocks of 100 and 51, and a font used for nothing it has still emits one entry. 100% statement coverage, go vet and -race clean, nine cross-compile targets. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A character the font does not have is written as glyph 0, .notdef. The first
such character also took glyph 0's /ToUnicode entry, and every later missing
character then read back as that first one — so the text a foreign reader
extracted was not text with holes in it but text that was confidently wrong,
which is the worse of the two failures.
Found by writing the lines a text writer gets wrong and letting poppler say
what it could read back. Written here, read by pdftotext:
a 漢字 b かな c 한글 d -> a 漢漢 b 漢漢 c 漢漢 d
السلام عليكم -> one Arabic letter, eleven times
a U+1F600 b U+1D400 c -> a U+1F600 b U+1F600 c
Every missing character in a document became whichever missing character came
first in it.
WHY THE REPLACEMENT CHARACTER RATHER THAN NOTHING
Two honest answers were available and both were measured rather than argued
about. Omitting the entry gives:
a 漢字 b かな c 한글 d -> abcd
السلام عليكم -> (nothing at all)
which loses the spaces along with the characters and leaves a reader no way to
know anything had been there. Mapping glyph 0 to U+FFFD gives:
a 漢字 b かな c 한글 d -> a __ b __ c __ d (U+FFFD each)
السلام عليكم -> eleven of them
The deciding measurement is what the page itself shows. Rendered at 72 dots to
the inch, the page draws 17 856 dark pixels either way: the face's own .notdef
draws a box, so the loss is already visible to a person looking at it. Text
read off that page should say what the page says — one mark per missing
character, in the right place — and U+FFFD is how Unicode says exactly that.
Nothing would have made the extraction disagree with the picture.
WHAT IS NOT AFFECTED
Both call sites are covered: Text and TextShaped mark through the same fontUse.
Everything the font does have is unchanged — Cyrillic, Greek, the punctuation a
producer substitutes, combining accents, astral-plane characters that the font
carries, and two thousand characters on one line all still read back exactly.
Ten files across five faces, plain and shaped, are parsed and drawn by poppler
with the text byte-identical to what was written, and pdffonts reports every
one as embedded, subsetted and with a Unicode map.
Two existing tests asserted the old contract and now assert the new one:
.notdef is always mapped, so 150 mapped glyphs make blocks of 100 and 51, and a
font used for nothing it has still emits one entry.
100% statement coverage, go vet and -race clean, nine cross-compile targets.