Add object streams and cross-reference streams to the writer - #7
Merged
Conversation
A PDF is mostly small dictionaries, and a dictionary written on its own cannot be compressed at all — there is nowhere to say how. Object streams are the format's answer: pack many objects into one stream and compress that. This adds them, and the cross-reference stream that is the only form able to say where a packed object lives. NewPackedWriter is the same Writer with that turned on. Objects of generation zero that are not themselves streams wait to be gathered up; streams are written where they stand, and one arriving with no filter of its own is deflated, since nothing this package generates comes compressed and a packed file is being written to be small. Two hundred objects go in each stream — one has to be inflated whole to read any object in it, so packing everything into one would make opening a file read all of it. The header is raised to 1.5, the version that introduced this, which is 2003. Measured on the corpus: all 118 833 files still rewrite to an identical fingerprint, and the output falls from 98.8% of the input to 94.6% — about 1.9 GB across the corpus. It also settles the 355 files that were coming out larger than they went in. The reader has been able to read both forms since its second wave, so this is the writer catching up with it — and the two now check each other on every file in the corpus. Exact 100% statement coverage, go vet clean, nine cross-compilation targets. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A PDF is mostly small dictionaries, and a dictionary written on its own cannot
be compressed at all — there is nowhere to say how. Object streams are the
format's answer: pack many objects into one stream and compress that. This
adds them, and the cross-reference stream that is the only form able to say
where a packed object lives.
NewPackedWriter is the same Writer with that turned on. Objects of generation
zero that are not themselves streams wait to be gathered up; streams are
written where they stand, and one arriving with no filter of its own is
deflated, since nothing this package generates comes compressed and a packed
file is being written to be small. Two hundred objects go in each stream — one
has to be inflated whole to read any object in it, so packing everything into
one would make opening a file read all of it. The header is raised to 1.5, the
version that introduced this, which is 2003.
Measured on the corpus: all 118 833 files still rewrite to an identical
fingerprint, and the output falls from 98.8% of the input to 94.6% — about 1.9
GB across the corpus. It also settles the 355 files that were coming out larger
than they went in.
The reader has been able to read both forms since its second wave, so this is
the writer catching up with it — and the two now check each other on every file
in the corpus.
Exact 100% statement coverage, go vet clean, nine cross-compilation targets.