Lo: Add 075/105 pivot angle background-subtracted ISN maps to dependencies

[jaredclaypoole]

Addresses IMAP-Science-Operations-Center/imap_L3_processing#179

Change Summary

Overview

Lo: Add 075/105 pivot angle background-subtracted ISN maps to dependencies

File changes

• sds_data_manager/orchestration/dependencies/imap_lo_dependencies.yaml
  • Add four L2 single-product jobs: l(075/105)-enanbs-(h/o)-sf-nsp-ram-hae-6deg-1yr
  • Add four L3 single-product jobs: l(075/105)-isn-(h/o)-sf-nsp-ram-hae-6deg-1yr

Testing

No changes to testing.

[jtniehof]

Sorry for the delay. Only question is whether we can actually make the off-90 L2 maps. @vineetbansal ?

[vineetbansal]

@jtniehof, @jaredclaypoole - only talking about the L2 maps, not the L3 (which I know nothing about) - we can and are already generating off-90 l2 maps.

This PR doesn't look right to me on several counts:

• It's adding both l2 and l3 entries to the dependencies at the same time - this should be done (and validated) in 2 different PRs.
• It's added these 4 l2 entries, but "enanbs" is not a valid descriptor as per the naming convention anyway (which I've followed in the existing l2 6mo maps). I doubt it will be processed correctly. I suspect it's been sitting in this yml for a while and has never worked.
  • l2, l075-enanbs-h-sf-nsp-ram-hae-6deg-1yr
  • l2, l075-enanbs-o-sf-nsp-ram-hae-6deg-1yr
  • l2, l105-enanbs-h-sf-nsp-ram-hae-6deg-1yr
  • l2, l105-enanbs-o-sf-nsp-ram-hae-6deg-1yr
• The PR is trying to add 1yr entries without ensuring that they're identical to the 6mo entries. My understanding is that the Lo team puts out the product list (and there are 28 of them, corresponding to the current 28 L2 6mo entries) - and the yml should ensure that the cadence varies as per project requirements.
• We're already able to do ISN masking in L2 maps (manifested in several Msk products). So I'd think we can produce ISN maps easily as well. Is there any reason this cannot/should not be done in L2? Might well be, just asking..

My suggestion would be to close this PR and open another one where we simply add the 28 1yr counterparts to the 28 6mo entries, make sure they're being processed correctly, and then proceed from there. What do you think?

[jaredclaypoole]

@vineetbansal I have no problem with closing this PR and will defer to you and Jon on that.

One question I have is whether this dependency yaml file should be auto-generated by some script rather than created by hand, especially if you're talking about adding 28 entries to it.

[vineetbansal]

No - I don't think writing a script for it is worth it. It's not like the product list comes from an upstream file readily available to us programmatically. Normally I'd copy the relevant portion of it in a buffer, find-replace the "6mo" with "1yr", and that should be fine.

Unless yml itself has a syntax to create sections where some sub-part of the key/value is variable and implicitly derived from other key/values (so that the yml reader in sds-data-manager is oblivious to the shortcut).

[jaredclaypoole]

I had AI generate a script for just the L2 descriptors for my own sanity. No need to make this permanent, but I'll include it here for the sake of communicating what's in the current yaml file:

def gen_l2_descriptors() -> list[str]:
    """Generate the supported Lo L2 descriptors in sorted order."""
    descriptors = []
    suffix = "nsp-ram-hae-6deg"

    # Six-month products share four product/frame pairs; ilo is masked only.
    six_month_pairs = (
        ("enansnbs", "sf"),
        ("enasbs", "hf"),
        ("enasbs", "sf"),
        ("enasnbs", "sf"),
    )
    for sensor in ("l075", "l090", "l105", "ilo"):
        masks = ("Msk",) if sensor == "ilo" else ("", "Msk")
        for mask in masks:
            for product, frame in six_month_pairs:
                descriptors.append(
                    f"{sensor}-{product}{mask}-h-{frame}-{suffix}-6mo"
                )

    # Annual products have sensor-specific product/species/frame combinations.
    annual_combinations = {
        "l090": (
            ("ena", "h", ("hf", "hk", "sf")),
            ("ena", "o", ("sf",)),
            ("enanbs", "h", ("hk", "sf")),
            ("enanbs", "o", ("sf",)),
            ("enansnbs", "h", ("sf",)),
            ("isn", "h", ("sf",)),
            ("isn", "o", ("sf",)),
        ),
        "t090": (
            ("ena", "h", ("sf",)),
            ("isn", "h", ("sf",)),
            ("isn", "o", ("sf",)),
        ),
    }
    for sensor, combinations in annual_combinations.items():
        for product, species, frames in combinations:
            for frame in frames:
                descriptors.append(
                    f"{sensor}-{product}-{species}-{frame}-{suffix}-1yr"
                )

    return sorted(descriptors)

A few questions for @vineetbansal:

• Invalid descriptors:
  • Is ena a valid descriptor for Lo? The document you linked says "For Lo: ena and spx will be followed by three sets of descriptors"
  • As you pointed out, enanbs is an invalid descriptor, according to the linked document
  • How should these invalid descriptors be corrected? Or should they be removed?
  • Note that l090enanbshhknspramhae6deg1yr has been materialized on Dagster, despite being an invalid descriptor: https://processing.imap-mission.com/runs/ccee846b-308b-4955-8e03-aa904e017517
• Adding additional 1yr jobs
  • You suggested adding 1yr counterparts of all 28 6mo maps. Currently there are 13 1yr maps, only one of which has the same descriptor (minus the 6mo/1yr part) as a 6mo map. (That one is l090-enansnbs-h-sf-nsp-ram-hae-6deg.) So would you like me to add the remaining 27 1yr maps and leave the other 12 1yr maps that have no corresponding 6mo descriptor? (Unless I'm missing something here.)

[vineetbansal]

@jaredclaypoole - my inclination would be to remove all current l2 1yr entries and start afresh, but let me get back to you tomorrow after consulting with the team.

For your questions:

Is ena a valid descriptor for Lo? The document you linked says "For Lo: ena and spx will be followed by three sets of descriptors"

• No - its invalid as per the document.

As you pointed out, enanbs is an invalid descriptor, according to the linked document

• Indeed

How should these invalid descriptors be corrected? Or should they be removed?

• Should be removed.

Note that l090enanbshhknspramhae6deg1yr has been materialized on Dagster, despite being an invalid descriptor: https://processing.imap-mission.com/runs/ccee846b-308b-4955-8e03-aa904e017517

• The regex used in https://github.com/IMAP-Science-Operations-Center/imap_processing/blob/dde076e9033dd14c2bd3efb345172b785c709864/imap_processing/ena_maps/utils/naming.py#L523 might be too permissive and allows it, but its still invalid as per the doc.

I'm using valid/invalid only going by the doc, not the code (which is common to all mapping instruments, not just Lo).