Add RFC for using Text-Services for PDF generation

[donaldgray]

What does this change?

Adds new RFC looking at how we can use text-services for PDF generation from NQs.

There are a couple of areas specifically called out for review, so RFC will need revisited once review has been carried out.

[tomcrane]

I think some of the problem/complexity is that to us it's still a PDF... it still looks exactly the same as the DLCS old version, at least at first. But it's actually a completely different thing - it's a textual thing that happens to have images. So it's not like the NQ it was before, which is just a (tweaked) projection of assets into a PDF. We need that too, but can't do it without also having something completely new, the protobuf word map which hasn't come from assets at all but has come from additional files, and we need that before we can start building the PDF; it's the new presence of text that drives the architecture change.

If DDS or Delft equivalent drove the text services API (ignoring for a moment how we access-control that), it could assemble the fireball-like job JSON sourceData with direct references to the ALTO files at s3: URIs, which don't ever need to be adjuncts. But Orchestrator doesn't know anything about those alto files in S3; if Orchestrator assembles the job JSON the only place it can learn about the ALTO files is if they have been registered as adjuncts on assets, that's all it knows about in DLCS-land, anything else is outside its knowledge. DLCS doesn't need to store them (their bytes), they can be external; but they do need to be rows in the adjuncts table. Whether orchestrator or DDS-equivalents drive construction is a trade-off, and Orchestrator wins I think because of the auth enforcement. Yes, we'll be storing asset->adjunct relationships in the DB even when we're assembling Manifests externally, like Wellcome or Delft but we have to convey the information somehow and this doesn't seem like a bad thing to have stored (and queryable).

In the Proposal sequence diagram, what is Orchestrator sending for "Upsert text-builder job" - the sourceData JSON or an actual manifest? I assume the former as it doesn't need to make a manifest, but it could be stated explicitly. Orchestrator is then using the iiifLink property of the asset adjuncts and their formats to decide whether to include them in the sourceData payload, echoing the logic TextServices uses to pull the appropriate assets out of a Manifest if you provide a Manifest. sourceData doesn't go through those rules, it's explicitly saying "use this for text".

"Find all images to be included" still has bespoke rules about what roles can be included and that access needs to be enforced on the generated PDF in scenarios where it has (eg) clickthrough assets present and visible.

[tomcrane]

TOCs

I'm not sure that the NQ can handle arbitrary-depth nested TOCs, and having to have matching strings getting back into the s2 field (for example) to capture the ranges seems very fragile, any one asset would break it.

If the job is supplied to text services as a manifest then obviously it can translate any ranges to the PDF TOC. And we could extend the fireball-like JSON job format sourceData to include a similar structures property that works like IIIF ranges. So TextServices can support TOC-creation in PDFs quite easily. But to assemble either of these from a NQ seems ... hard. Seems like it would only work for iiif-p hosted manifests where a manifest-level PDF adjunct could be created from the manifest directly.

How would Wellcome or similar non iiif-p PDFs get arbitrary structure information into the named query output without the groupBy mechanism? It needs to delegate this to something that knows more than the DLCS does.

[tomcrane]

(very sketchy idea coming up... not saying it's a good idea)

assetOrder=n1;n2&s1=p1&structureProvider=https://example.org/service/structureProvider

if there's a structureProvider, once orchestrator has assembled the asset list it POSTS it to the service and it returns a suitable structures property... eek... ugh

{
   "assetOrder": "n1,n2",
   "s1": "p1",
   "p1": "b1234",   // hydrated, specific
   "assets": [
      "2/5/image1",
      "2/5/image2",
      "2/5/image3",
      "2/5/image4",
      "2/5/image5",
   ]
}

service receives this, asks itself "OK what's the structure arrangement for the assets in b1234?"

returns

{
   "assetOrder": "n1,n2",
   "s1": "p1",
   "p1": "b1234",  
   "toc": [
        {
             "label": "Table of Contents",
             "items": [
                   {
                       "label": "Glasgow",
                       "assets": [
                           "2/5/image1",
                           "2/5/image2",
                           "2/5/image3",
                        ]
                   },
                   {
                       "label": "London",
                       "assets": [
                           "2/5/image4",
                           "2/5/image5"
                        ]
                   }
              ]
        }
   ]
} 

hmm...

[donaldgray]

@tomcrane - addressed issues in your comments. Changes

• Called out that Orchestrator will construct the payload, not point text-builder at Manifest.
• Added structureProvider NQ parameter, similar idea but slightly different from your suggestion.

There's still a couple of outstanding "For PR reviewers" comments that we might be better to discuss, or the correct answer might come out of reviews.

[tomcrane]

Approving the PR while leaving some questions unresolved - talk to user(s)!

[tomcrane]

This bit is synchronous in text services? If it hasn't already built the PDF it will do it at this point. TextSearch API generates it lazily on first request and then stores it, and it may take some time, so a potentially lengthy activation should be shown here I think - or an alt box.

[tomcrane]

...agree useful, would you just do this if RFC approved?

[tomcrane]

It would have to carry through to different job IDs and different saved artefacts in Text Services - and would it lead to many variants of the protobuf textmaps being stored, one ofr each pdf permutation? With more cleverness it could only create one textmap and then different PDFs from it but even then it puts a lot of storage overhead. I don't think this option should be exposed to end users for sure.

[tomcrane]

If it can be deferred cleanly then let's leave this out, for now. Even if we revisit immediately.

[tomcrane]

Should be clear that the fields are existing asset props, e.g., string3

[tomcrane]

I definitely favour structureProvider as we stuck as soon as someone wants a two-layer TOC, and the metadata management seems messy to me. But we should check what a potential client user of this can actually support.

[tomcrane]

Inclined to defer as mentioned above