Skip to content

.notdef must not claim to be a character - #7

Merged
tannevaled merged 1 commit into
mainfrom
fix/notdef-must-not-claim-a-character
Aug 27, 2026
Merged

.notdef must not claim to be a character#7
tannevaled merged 1 commit into
mainfrom
fix/notdef-must-not-claim-a-character

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

A character the font does not have is written as glyph 0, .notdef. The first
such character also took glyph 0's /ToUnicode entry, and every later missing
character then read back as that first one — so the text a foreign reader
extracted was not text with holes in it but text that was confidently wrong,
which is the worse of the two failures.

Found by writing the lines a text writer gets wrong and letting poppler say
what it could read back. Written here, read by pdftotext:

a 漢字 b かな c 한글 d -> a 漢漢 b 漢漢 c 漢漢 d
السلام عليكم -> one Arabic letter, eleven times
a U+1F600 b U+1D400 c -> a U+1F600 b U+1F600 c

Every missing character in a document became whichever missing character came
first in it.

WHY THE REPLACEMENT CHARACTER RATHER THAN NOTHING

Two honest answers were available and both were measured rather than argued
about. Omitting the entry gives:

a 漢字 b かな c 한글 d -> abcd
السلام عليكم -> (nothing at all)

which loses the spaces along with the characters and leaves a reader no way to
know anything had been there. Mapping glyph 0 to U+FFFD gives:

a 漢字 b かな c 한글 d -> a __ b __ c __ d (U+FFFD each)
السلام عليكم -> eleven of them

The deciding measurement is what the page itself shows. Rendered at 72 dots to
the inch, the page draws 17 856 dark pixels either way: the face's own .notdef
draws a box, so the loss is already visible to a person looking at it. Text
read off that page should say what the page says — one mark per missing
character, in the right place — and U+FFFD is how Unicode says exactly that.
Nothing would have made the extraction disagree with the picture.

WHAT IS NOT AFFECTED

Both call sites are covered: Text and TextShaped mark through the same fontUse.
Everything the font does have is unchanged — Cyrillic, Greek, the punctuation a
producer substitutes, combining accents, astral-plane characters that the font
carries, and two thousand characters on one line all still read back exactly.
Ten files across five faces, plain and shaped, are parsed and drawn by poppler
with the text byte-identical to what was written, and pdffonts reports every
one as embedded, subsetted and with a Unicode map.

Two existing tests asserted the old contract and now assert the new one:
.notdef is always mapped, so 150 mapped glyphs make blocks of 100 and 51, and a
font used for nothing it has still emits one entry.

100% statement coverage, go vet and -race clean, nine cross-compile targets.

A character the font does not have is written as glyph 0, .notdef. The first
such character also took glyph 0's /ToUnicode entry, and every later missing
character then read back as that first one — so the text a foreign reader
extracted was not text with holes in it but text that was confidently wrong,
which is the worse of the two failures.

Found by writing the lines a text writer gets wrong and letting poppler say
what it could read back. Written here, read by pdftotext:

  a 漢字 b かな c 한글 d   ->   a 漢漢 b 漢漢 c 漢漢 d
  السلام عليكم            ->   one Arabic letter, eleven times
  a U+1F600 b U+1D400 c   ->   a U+1F600 b U+1F600 c

Every missing character in a document became whichever missing character came
first in it.

WHY THE REPLACEMENT CHARACTER RATHER THAN NOTHING

Two honest answers were available and both were measured rather than argued
about. Omitting the entry gives:

  a 漢字 b かな c 한글 d   ->   abcd
  السلام عليكم            ->   (nothing at all)

which loses the spaces along with the characters and leaves a reader no way to
know anything had been there. Mapping glyph 0 to U+FFFD gives:

  a 漢字 b かな c 한글 d   ->   a __ b __ c __ d      (U+FFFD each)
  السلام عليكم            ->   eleven of them

The deciding measurement is what the page itself shows. Rendered at 72 dots to
the inch, the page draws 17 856 dark pixels either way: the face's own .notdef
draws a box, so the loss is already visible to a person looking at it. Text
read off that page should say what the page says — one mark per missing
character, in the right place — and U+FFFD is how Unicode says exactly that.
Nothing would have made the extraction disagree with the picture.

WHAT IS NOT AFFECTED

Both call sites are covered: Text and TextShaped mark through the same fontUse.
Everything the font does have is unchanged — Cyrillic, Greek, the punctuation a
producer substitutes, combining accents, astral-plane characters that the font
carries, and two thousand characters on one line all still read back exactly.
Ten files across five faces, plain and shaped, are parsed and drawn by poppler
with the text byte-identical to what was written, and pdffonts reports every
one as embedded, subsetted and with a Unicode map.

Two existing tests asserted the old contract and now assert the new one:
.notdef is always mapped, so 150 mapped glyphs make blocks of 100 and 51, and a
font used for nothing it has still emits one entry.

100% statement coverage, go vet and -race clean, nine cross-compile targets.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit a7ba71c into main Aug 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant