Carry the reading order a screen reader follows

[tannevaled]

A tagged document says which run of marks on which page is a heading, a
paragraph, a table cell, the label of a form field. That is the reading order a
screen reader follows, and for a government form it is often what the law
requires — RGAA in France, Section 508 in the United States, EN 301 549 in the
EU. #11 rebuilt the catalogue around the pages that survived and deliberately
left /StructTreeRoot behind, because a tree that points at the wrong pages is
worse than none: a reader follows it into the wrong order instead of falling
back on the text. This rebuilds it.

Before: what is actually there

Over the 1 633 real forms in /Users/Shared/pdfforms (the 635 vendor
fixtures excluded):

carry a /StructTreeRoot	1 021
of those, carry a /ParentTree	1 012
carry /RoleMap / /ClassMap / /IDTree	984 / 356 / 173
pages saying where their marks are filed	6 869 of 7 114
structure elements	853 108
marks by number / by reference / annotation references	470 269 / 735 / 104 521
empty elements (30 345 of them empty table cells)	39 350

The finding that shaped the design

A mark is filed under a key the object holding it carries, and it is not
always the page. Of the 1 021 tagged forms:

• 991 file every mark under a page;
• 14 file some under a page and some under a form XObject drawn on it;
• 3 — the UK lasting-power-of-attorney forms — carry no key on any page at
  all and file everything under the XObjects, whose stream dictionaries say
  /StructParents 0, 1, 2, 3 while every page of the file says nothing.

6 869 keys sit on pages, 150 on drawn streams. That is read off the files, not
off a specification, and it settles two things. A mark inside a drawn stream
cannot be filed under the page's number — the two are numbered in different
spaces. And the numbers handed to pages must start above every key a drawn
stream already carries, because a copied stream keeps its key: a stream is
written before the structure above it is rebuilt, so it cannot be given a new
one. Numbering pages from zero would eventually hand a page the number a form
XObject on it is filed under, and a reader looking up a mark in that XObject
would be told, with every confidence, about the page's own elements.

What is carried

The tree of elements, each element's page, the marks inside those pages,
marked-content references, references to annotations, /RoleMap, /ClassMap,
/IDTree, and the number tree rebuilt in both directions. Numbers are handed
out afresh. An element whose every page has gone is removed and its parent
pruned; one with some pages gone keeps the children that are left; an empty
table cell is kept, because the shape of a table is part of what it says. A page
written twice carries the structure once, on the first copy, since an element
names one page.

After: rotating all 1 633 forms

1 021 trees in, 1 014 out. The seven that lose theirs are the seven whose
/StructTreeRoot has no /K in the file we were given — a root with a role map
and nothing under it. 991 of the 1 021 come out identical down to the last
element, mark, reference and annotation reference. The thirty that differ are
all of them:

• 23 files drop 142 references to annotations that are on no page of
  the source either; 104 379 of 104 521 point at an annotation on a page. Ten of
  the 23 lose 236 elements with them — the ones whose only child was such a
  reference.
• 4 files drop 109 marks inside a stream no surviving page draws. 109 of
  the corpus's 215 such marks named a stream no page of the source drew either;
  the other 106 are carried, with the stream, under the key the stream
  carries. Two power-of-attorney forms go from 158 elements to 91 this way,
  cerfa_12496 from 16 to 10.
• 3 files drop 33 marks with no page anywhere above them.

Nothing else moves. /RoleMap 984 → 977, /ClassMap 356 → 349, /ParentTree
1 012 → 1 004: in each case the missing ones are the seven empty trees, plus one
file whose tree holds no mark and no annotation, so there is nothing to file.
/IDTree 173 → 153: the 20 files that lose it carry no /ID on any element, so
their identifier tree named nothing.

Reading order, not just presence

The check walks both trees in step. At every level the children left must be the
children that were there, in the order they were there, with nothing added
and nothing moved; elements are told apart by what they contain, and the pairing
backtracks — two sibling table cells of the same kind are indistinguishable
until their contents are compared, and a checker that commits to the first
plausible pairing reports a defect of its own.

Then on the output alone:

check	failures
every element's /Pg names a page that exists	0
every mark named is a mark that page's content stream draws	0 new
number tree agrees with the tree, both directions, for pages	0
ditto for marks inside drawn streams	0 new
every annotation's /StructParent names its owning element	0
pages left filed under a number the tree does not use	0
cycles reached	0

The output names 1 387 marks the page does not draw — the source names the
same 1 387, in the same 17 files, mark for mark. Not one wrong pointer is
introduced. On the number tree we come out ahead of the source: 4 stream marks
are misfiled in the file we were given and 1 in ours, the one remaining being
two form XObjects that share a key and both claim mark 0, which a number tree
can only answer once.

The same check over five other verbs

• Keeping the odd pages: 1 009 of 1 021 trees. The five extra losses are
  files whose entire marked content is on even pages, so keeping the odd ones
  leaves nothing to describe. 210 191 marks and 49 302 annotation references
  pruned with their pages, 358 261 elements removed, 0 unexplained.
• Writing a page twice: identical to the rotate, and the second copy carries
  no /StructParents.
• Flatten and remove-annotations: all 104 521 annotation references go
  and 179 435 elements with them; not one dangling reference is left.
• Watermark: identical to the rotate — a stamp does not disturb the marks
  underneath it.
• Merge: no tree at all, and the 6 897 pages that said where their structure
  was filed now say nothing rather than pointing into a tree that is not there.

Two defects this found

• A page written twice left anything pointing at its annotations pointing at the
  last copy rather than the first — for a structure tree, describing page
  one and sending a reader to a widget on page two. The annotations a rebuild
  keeps now answer with the first copy, the same choice the destination map
  already made for the page itself.
• An element whose own page was dropped but which survived through a child on
  another page was left with no /Pg, and so silently inherited the page of
  whatever it sat under — a page it is not on, stated as confidently as the
  right one would have been. It now names the page its remaining content is on.

Not carried, with the reason in catalogue.go

A mark inside a stream no surviving page draws; a structure element's /Ref
(PDF 2.0, no file in the corpus has one, and copying it would drag a second copy
of the source's tree and pages in behind it); and the structure of pages from
more than one file — two files have two /RoleMap and /ClassMap dictionaries
in which the same name may stand for two different things, and there is no
honest way to choose.

No regression

#11's figures reproduce exactly. Over the whole corpus of 2 268: 1 338 of
1 340 forms survive a rotate, 1 335 with every field, the same three files
losing the same fields; language 866 of 866, marked-up flag 1 057 of 1 057.
Checked by running the same harness against v0.6.0, which carries 0 trees of the
1 105 the corpus offers.

Exact 100% statement coverage, go vet clean, -race clean, cross-compiles for
all nine targets. No new dependency; reader stays at v0.4.1, which has
everything this needs.