Keep what a document says about itself, and its form - #11
Merged
Conversation
A document is more than its pages. Its catalogue says what language it is in,
whether its structure has been marked up for a screen reader, how a viewer
should open it — and, if it has one, where its form is.
Every verb here rebuilds the document around the pages it kept, and the
catalogue was rebuilt as two entries: the page tree and the word Catalog.
Everything else went. Rotating a tax return kept all 199 of its widget
annotations on the pages and threw away the field list that gives them
meaning — not a form with something missing but half a form, which is worse
than none, since every widget is still there to be seen and none of them can
be filled in. The language and the marked-up flag went with it, which for a
government form is not merely untidy.
The field list is now rebuilt around the widgets that survived. A field merged
into its own widget — how nearly every field with one place on the page is
written — is already in the output, so it is pointed at rather than copied
again; a field with widgets of its own is written afresh with its children
pointing back at it, however deep the tree goes.
Measured over the corpus of 2 268 real forms: 1 338 of 1 339 forms survive a
rotate, 1 335 of them with every field, no panics. Every difference is
accounted for:
- The fields that go are exactly those whose widgets are on no page at all,
which could never have been seen or filled in the original either. One US
courts form has 179 of them beside its 156 real ones, and 156 plus 179 is
what it had.
- The one form that disappears entirely is qpdf's own remove-annotations
output — a file that was already a half-form when we got it, with all three
of its fields pointing at widgets nobody had left on a page.
Language kept 556 of 556, marked-up flag 738 of 738.
Pages from several files keep their pages and nothing above them: two files
have two catalogues and two forms whose fields may be named the same, and there
is no honest way to choose between them or to merge them.
What is still not carried is named in catalogue.go with the reason for each.
The largest is /StructTreeRoot, which names the pages and the numbered marks
inside their content and needs a rebuild of its own — and which a screen reader
would follow into the wrong order if it were carried across wrong.
Found by the robustness campaign.
tannevaled
force-pushed
the
keep-the-catalogue
branch
from
August 27, 2026 08:13
57e423a to
ae95607
Compare
tannevaled
added a commit
that referenced
this pull request
Aug 27, 2026
A tagged document says which run of marks on which page is a heading, a paragraph, a table cell, the label of a form field. That is the reading order a screen reader follows, and for a government form it is often what the law requires: RGAA in France, Section 508 in the United States, EN 301 549 in the EU. #11 rebuilt the catalogue around the pages that survived and deliberately left /StructTreeRoot behind, because carrying it wrong sends a reader through the document in the wrong order rather than letting it fall back on the text. It is now rebuilt. Of 1 633 real forms from eighteen issuers, 1 021 carry a structure tree and 1 012 of those carry the number tree that indexes it; 6 869 of their 7 114 pages say under what number their marks are filed. The tree holds 853 108 elements, 470 269 marks named by number, 735 named by reference and 104 521 references to annotations. WHERE THE KEYS LIVE. A mark is filed under a key the object holding it carries, and it is not always the page. Of the 1 021 tagged forms, 991 file every mark under a page; 14 file some under a page and some under a form XObject drawn on it; and 3 — the UK lasting-power-of-attorney forms — carry no key on any page at all and file everything under the XObjects, whose stream dictionaries say /StructParents 0, 1, 2 and 3 while every page of the file says nothing. 6 869 keys sit on pages and 150 on drawn streams. That is read off the files, not off a specification, and it settles two things: a mark inside a drawn stream cannot be filed under the page's number, and the numbers handed to pages must start above every key a drawn stream already carries — because a copied stream keeps its key, a stream being written before the structure above it is rebuilt. WHAT IS CARRIED. The tree of elements, each element's page, the marks inside those pages, marked-content references, references to annotations, /RoleMap, /ClassMap, /IDTree, and the number tree, rebuilt in both directions. The numbers are handed out afresh: a page that kept the number it had in a file it is no longer part of is a page a reader would look up and be told, with every confidence, about somebody else's. An element whose every page has gone is removed and its parent pruned; one with some pages gone keeps the children that are left; an empty table cell is kept, because the shape of a table is part of what it says, and the corpus has 30 345 of them. A page written twice carries the structure once, on the first copy, since an element names one page. MEASURED. Rotating all 1 633: 1 021 trees in, 1 014 out. The seven that lose theirs are the seven whose /StructTreeRoot has no /K in the file we were given — a root with a role map and nothing under it. 991 of the 1 021 come out identical down to the last element, mark, reference and annotation reference. The thirty that differ are all of them: - 23 files drop 142 references to annotations that are on no page of the source either — the same 142 the OBJR survey found; 104 379 of 104 521 point at an annotation that is on a page. Ten of the 23 lose 236 elements with them, the elements whose only child was such a reference. - 4 files drop 109 marks inside a stream that no surviving page draws. 109 of the corpus's 215 such marks named a stream no page of the source drew either; the other 106 are carried, with the stream, under the key the stream carries. Two UK power-of-attorney forms go from 158 elements to 91 this way and cerfa_12496 from 16 to 10. - 3 files drop 33 marks that have no page anywhere above them. Nothing else moves: /RoleMap 984 in and 977 out, /ClassMap 356 and 349, /ParentTree 1 012 and 1 004 — in each case the missing ones are the seven empty trees, plus one file whose tree holds no mark and no annotation, so there is nothing to file. /IDTree goes from 173 to 153: the 20 files that lose it have no element carrying an /ID at all, so their identifier tree named nothing. READING ORDER, NOT JUST PRESENCE. The check walks both trees in step. At every level the children left must be the children that were there, in the order they were there, with nothing added and nothing moved; elements are told apart by what they contain, and the pairing backtracks, because two sibling table cells of the same kind are indistinguishable until their contents are compared and a checker that commits to the first plausible pairing reports a defect of its own. Then, on the output alone: every element's /Pg names a page that exists (0 failures), every mark the tree names is a mark that page's content stream really draws, and the number tree agrees with the tree in both directions — every mark the tree claims is filed under the page's key, and every non-null place in the page's array is a mark the tree claims (0 failures), and the same for the annotations (0 failures). The output names 1 387 marks that the page does not draw. The source names the same 1 387, in the same 17 files, mark for mark: not one wrong pointer is introduced. On the number tree we come out ahead of the source — 4 stream marks are misfiled in the file we were given and 1 in ours, the one that remains being two form XObjects that share a key and both claim mark 0, which a number tree can only answer once. The same check over five other verbs. Keeping the odd pages: 1 009 trees out of 1 021 — the five extra losses are files whose entire marked content is on even pages, so keeping the odd ones leaves nothing to describe — 210 191 marks and 49 302 annotation references pruned with their pages, 358 261 elements removed, 0 unexplained. Writing a page twice: identical to the rotate, and the second copy carries no /StructParents. Flatten and remove-annotations: all 104 521 annotation references go and 179 435 elements with them, and not one dangling reference is left. Watermarking: identical to the rotate, so a stamp does not disturb the marks underneath it. Merging two files carries no tree, as the catalogue and the form already did: two files have two /RoleMap and /ClassMap dictionaries in which the same name may stand for two different things, and there is no honest way to choose. The 6 897 pages that said where their structure was filed now say nothing, rather than pointing into a tree that is not there. TWO DEFECTS THIS FOUND. A page written twice used to leave anything pointing at its annotations pointing at the last copy rather than the first, which for a structure tree means describing page one and sending a reader to a widget on page two; the annotations a rebuild keeps now answer with the first copy, the same choice the destination map already made for the page itself. And an element whose own page was dropped but which survived through a child on another page was left with no /Pg, and so silently inherited the page of whatever it sat under — a page it is not on, stated as confidently as the right one would have been. It now names the page its remaining content is actually on. NOT CARRIED, with the reason in catalogue.go: a mark inside a stream no surviving page draws, an element's /Ref, and the structure of pages from more than one file. #11's figures do not move. Over the whole corpus of 2 268: 1 338 of 1 340 forms survive a rotate, 1 335 with every field, the same three files losing the same fields; language 866 of 866, marked-up flag 1 057 of 1 057. Checked by running the same harness against v0.6.0, which carries 0 trees of the 1 105 the corpus offers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A document is more than its pages. Its catalogue says what language it is in, whether its structure has been marked up for a screen reader, how a viewer should open it — and, if it has one, where its form is.
Every verb here rebuilds the document around the pages it kept, and the catalogue was rebuilt as two entries: the page tree and the word
Catalog.Rotating a tax return kept all 199 of its widget annotations on the pages and threw away the field list that gives them meaning. That is not a form with something missing — it is half a form, and worse than none: every box is still there to be seen and not one of them can be filled in.
What now happens
The field list is rebuilt around the widgets that survived. A field merged into its own widget — how nearly every field with one place on the page is written — is already in the output, so it is pointed at rather than copied again; a field with widgets of its own is written afresh with its children pointing back at it, however deep the tree goes.
Measured over the 2 268-form corpus
Every difference is accounted for, which is the part that matters:
/Annotsis never drawn and cannot be clicked.us-uscourts/ao187.pdfhas 179 of those beside its 156 real ones, and 156 + 179 = 335, which is what it had.remove-annotationsoutput — a file that was already a half-form when we got it, all three of its fields pointing at widgets nobody had left on a page. We clean it up rather than propagate it.What is deliberately not carried
Pages from several files keep their pages and nothing above them: two files have two catalogues and two forms whose fields may be named the same, and there is no honest way to choose or to merge.
catalogue.gonames each dropped entry with its reason. The largest is/StructTreeRoot, the marked-up structure a screen reader follows: its elements name the page each belongs to and the numbered marks inside that page's content, and its parent tree is indexed by a number the page carries. Carrying it means rebuilding all three, and a structure tree pointing at the wrong pages is worse than none — a reader would read the document aloud in the wrong order rather than fall back on the text. That is the one worth doing next.Found by the robustness campaign. 100% statement coverage in both the package and the command,
go vetandgofmtclean, nine cross-compile targets.