Skip to content

Add the content-stream tokeniser and inline images - #4

Merged
tannevaled merged 1 commit into
mainfrom
content-streams
Aug 25, 2026
Merged

Add the content-stream tokeniser and inline images#4
tannevaled merged 1 commit into
mainfrom
content-streams

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

A page's marking operations, and the one thing in a content stream that cannot
be read by looking at it. Operators come back with their operands, and rubbish
in the middle costs the byte it occupies rather than everything after it, which
is what a renderer needs from a corpus where a handful of producers write bytes
no grammar allows.

Where an inline image's data stops is genuinely ambiguous: the bytes between ID
and EI are raw, and compressed bytes do spell EI by accident. Three answers are
tried in order of how far they can be trusted. An image with no filter states
its own length exactly through width, height, depth and colour space. A
declared /L is believed when an EI really does follow it. Failing both, every
EI in the data is tried and the first whose data actually decodes — through the
declared filter, to the number of samples the image claims, and for a JPEG with
the markers a JPEG begins and ends with — is the one taken. Scanning alone gets
this wrong, and the corpus proves it: one page of 1023 inline images turned
into binary rubbish until the candidates were made to decode before being
believed.

The number lexer now stops at the first byte that cannot continue a number
rather than swallowing the whole run of regular characters. Producers write
"3.4-5" for two numbers, and twenty-one pages in the corpus were losing
everything after that point. The same change makes "12x" two tokens instead of
an error, which is what every other reader does.

Measured over the whole corpus: 1 536 769 753 operations across 138 337 pages
in ninety seconds, no panics, no page failing to decode, 15 854 inline images
read, and seventy distinct operators — every one of them real, where before
these two fixes there were ninety-nine and the extra twenty-nine were image
data being read as code.

Exact 100% statement coverage, go vet clean, nine cross-compilation targets.

A page's marking operations, and the one thing in a content stream that cannot
be read by looking at it. Operators come back with their operands, and rubbish
in the middle costs the byte it occupies rather than everything after it, which
is what a renderer needs from a corpus where a handful of producers write bytes
no grammar allows.

Where an inline image's data stops is genuinely ambiguous: the bytes between ID
and EI are raw, and compressed bytes do spell EI by accident. Three answers are
tried in order of how far they can be trusted. An image with no filter states
its own length exactly through width, height, depth and colour space. A
declared /L is believed when an EI really does follow it. Failing both, every
EI in the data is tried and the first whose data actually decodes — through the
declared filter, to the number of samples the image claims, and for a JPEG with
the markers a JPEG begins and ends with — is the one taken. Scanning alone gets
this wrong, and the corpus proves it: one page of 1023 inline images turned
into binary rubbish until the candidates were made to decode before being
believed.

The number lexer now stops at the first byte that cannot continue a number
rather than swallowing the whole run of regular characters. Producers write
"3.4-5" for two numbers, and twenty-one pages in the corpus were losing
everything after that point. The same change makes "12x" two tokens instead of
an error, which is what every other reader does.

Measured over the whole corpus: 1 536 769 753 operations across 138 337 pages
in ninety seconds, no panics, no page failing to decode, 15 854 inline images
read, and seventy distinct operators — every one of them real, where before
these two fixes there were ninety-nine and the extra twenty-nine were image
data being read as code.

Exact 100% statement coverage, go vet clean, nine cross-compilation targets.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tannevaled
tannevaled merged commit c337af3 into main Aug 25, 2026
1 check passed
@tannevaled
tannevaled deleted the content-streams branch August 25, 2026 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant