1444 bug pointing attitude job is not being run

[tmplummer]

Fix and enable spacecraft pointing-attitude processing job

Background

The spacecraft/l1a/pointing-attitude job produces a SPICE kernel encoding
despun spacecraft attitude for each pointing period. Its inputs are an
attitude_history (ah) SPICE kernel and a repoint file; it has no science
data inputs.

The ah kernel has an unusual delivery pattern. Each new delivery appends
coverage to the previous file, growing the covered time range up to
approximately three months before resetting. Early in the mission, before the
appending scheme was adopted, ah files covered approximately one day each.
When conops changed, the team retroactively produced a single combined file
covering the first ~three months. Because spice_files.file_name is unique,
none of those superseded files are ever deleted — they remain in the DB
alongside the kernel(s) that replaced them.

Because one run of the pointing-attitude job corresponds to the full coverage
of one ah kernel cycle — not a calendar day — the partition and sensor logic
require special handling compared to other processing jobs.

---

Changes

Bug fixes in imap_job.py

1. Science-input guard was unconditional

get_science_files_inputs raised
MissingDependenciesError("All jobs require at least one science file.")
whenever the collected science inputs were empty — even when the job was not
configured with any science inputs at all. The pointing-attitude job has only
SPICE and repoint inputs, so it hit this guard on every run. Fixed by adding
and self.job_config.science_inputs to the condition, so the error is only
raised when science inputs were expected but not found.

2. Sensor had no handler for repoint dependency type

The build_sensor loop dispatches on dependency.data_type to determine
which table to query for new files. The supported types were
VALID_DATALEVELS, spice, spin, and ancillary. There was no repoint
case, so new repoint files never produced any target_partitions and never
triggered the pointing-attitude job. Added an
elif dependency.data_type == "repoint": branch calling
trigger_from_new_non_science_inputs with models.RepointFiles.

3. RepointFiles has no start_date column

trigger_from_new_non_science_inputs defaults to reading start_date and
end_date from each file record. RepointFiles only has end_date. Without
explicit column overrides, this would raise AttributeError on the second
sensor run (after the cursor advances past MISSION_START_TIME). The repoint
branch passes datetime_start_column="end_date" and
datetime_end_column="end_date" to avoid this.

---

New custom partition: pointing_attitude (custom_partitions.py)

daily partitions were previously configured for this job, meaning the
sensor would attempt to run once per calendar day. This was wrong: the output
of each job run covers the full time range of the ah kernel (potentially
months), not a single day. One run corresponds to one ah kernel cycle.

The new pointing_attitude_partitions (DynamicPartitionsDefinition)
creates one partition per ah kernel, with the key encoding the pointing
times covered — not the raw kernel timestamps:

• Start: pointing_start_utc of the first pointing with any overlap with the ah kernel's coverage
• End: pointing_end_utc of the last pointing completely contained within the ah kernel's coverage

Using pointing times (rather than kernel timestamps) aligns the partition
boundaries exactly with the science data being processed and ensures no
partially-covered pointing is claimed as complete.

imap_spacecraft_dependencies.yaml's (l1a, pointing-attitude) entry is
updated to partition: pointing_attitude (was repoint), wiring the job to
this new partition set instead of the repoint-keyed one. The trigger for the
job remains the repoint file, since it's delivered to the SDC after the
updated ah kernel and is therefore the reliable signal that all inputs are
ready — only the partitioning changed, not the trigger.

Partition lifecycle — subsumption-based deletion

The add_pointing_attitude_partitions sensor handles two update scenarios by
deleting any existing partition whose full range is contained within the new
partition's range before adding the new one:

• Growing-append case: a new ah delivery extends the end date of an existing cycle. The old partition (same start, earlier end) is subsumed and replaced.
• Retroactive combined-file case: a single large file covers a span previously represented by many small daily partitions. All of the older smaller partitions are subsumed and deleted in one sensor run, replaced by the single combined partition.

Superseded kernels are filtered out before partitions are generated

Because old ah kernel rows are never deleted, the sensor previously kept
regenerating partitions for kernels whose coverage was fully contained within
a newer combined/append kernel — undoing the subsumption-based deletion above
(a partition gets deleted as subsumed, then immediately re-added because its
source kernel is still in the DB with no matching partition). A new
_select_maximal_ah_kernels helper filters attitude_kernels down to only
those not fully contained within another kernel's coverage before any
partitions are computed.

Kernels are sorted by coverage duration (longest first) so each candidate is
only compared against the kernels already accepted as maximal — that list
stays small in practice (one entry per "generation" of combined files),
keeping this cheap even on a cold start against a hundred-plus kernels. As a
side effect, partitions_to_add now lists larger-duration partitions first.

Partition key prefix

The prefix pointingattitude (no underscores) is required because
parse_dates_from_partition_key splits on the first _ to isolate the date
range portion of the key. A multi-word prefix with underscores would leave
part of the prefix attached to the start datetime string, causing strptime
to fail.

---

Tests (tests/orchestration/test_custom_partitions.py)

Nine unit tests for add_pointing_attitude_partitions, added to the existing
test_custom_partitions.py (alongside the idex/cadence sensor tests already
there) rather than a standalone file:

• No attitude history kernels in the database (no-op)
• No pointings overlapping the ah kernel coverage
• Partial coverage only — first_overlapping exists but last_covered is None
• New partition created on first run
• Already up to date — exact partition key already registered (no-op)
• Growing-append case — old partition replaced when end date extends
• Retroactive combined-file case — multiple daily partitions subsumed and replaced
• A kernel fully contained within another kernel's coverage is dropped and never produces its own partition
• Overlapping-but-not-nested kernels are both kept and processed longest-duration-first, regardless of input order

---

pyproject.toml

Added -p no:anyio to addopts. Installing the cdk-install dependency
group (which includes dagster) also pulls in anyio, which registers a
pytest plugin requiring pytest ≥7. The project uses pytest 6.2.5. Disabling
the plugin via -p no:anyio restores normal test collection without
requiring a pytest upgrade.

[Copilot]

Pull request overview

Fixes and enables the IMAP spacecraft spacecraft/l1a/pointing-attitude processing job by ensuring it can be triggered from its non-science inputs and by introducing a custom dynamic partitioning scheme aligned to attitude-history (AH) kernel coverage.

Changes:

• Add repoint dependency handling in the kickoff sensor and fix an unconditional “missing science inputs” guard for jobs that have no science inputs.
• Introduce a new pointing_attitude dynamic partition definition + sensor to maintain partitions based on AH kernel coverage mapped onto pointing times.
• Update the spacecraft job dependency config to use the new partition type, and adjust pytest options to avoid the anyio plugin incompatibility.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
sds_data_manager/orchestration/imap_job.py	Adds repoint dependency triggering and fixes science-input handling for non-science jobs.
sds_data_manager/orchestration/dependencies/imap_spacecraft_dependencies.yaml	Switches pointing-attitude from daily to pointing_attitude partitions.
sds_data_manager/orchestration/custom_partitions.py	Adds the pointing_attitude_partitions dynamic partition + sensor and registers it.
pyproject.toml	Disables the anyio pytest plugin to keep pytest 6.x collection working.

Comments suppressed due to low confidence (1)

sds_data_manager/orchestration/imap_job.py:843

• This error message is now triggered only when science_inputs are configured, but it still says “All jobs require at least one science file.” That’s misleading (and contradicts the existence of non-science-only jobs like pointing-attitude). Update the message to reflect the actual condition: science inputs were configured/required but none were found.

            raise MissingDependenciesError(
                "No science files were discovered. "
                "All jobs require at least one science file."
            )

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

🟡 Changes recommended

The new pointing-attitude partition sensor has a verified inconsistency between its “fully covered” pointing filter and the partition end timestamp, which can generate partitions that claim uncovered time ranges.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

sds_data_manager/orchestration/custom_partitions.py:420

• last_covered is constrained by repoint_start_utc <= ah_max, but the partition end is taken from pointing_end_utc. In the pointing table, pointing_end_utc is derived from the next row’s repoint_end_utc (see spice_indexer.index_pointing_data), so this can produce a partition end timestamp that extends beyond the attitude-history kernel coverage window implied by the filter. Align the containment filter with the timestamp used in the partition key to avoid generating partitions that claim uncovered time.

            # Last pointing completely contained within the ah kernel coverage
            last_covered = (
                session.query(models.PointingTable)
                .filter(
                    models.PointingTable.pointing_start_utc >= ah_min,
                    models.PointingTable.repoint_start_utc <= ah_max,
                )
                .order_by(models.PointingTable.pointing_end_utc.desc())
                .first()

• Files reviewed: 5/5 changed files
• Comments generated: 2
• Review effort level: Lite

[jaredclaypoole]

A few small things, otherwise this looks good.

[jaredclaypoole]

Thanks, LGTM now!

[bryan-harter]

Thanks so much for this! I think this should really help with cutting down on assets getting triggered. Not to mention it is just a way more accurate way of viewing the pointing_attitude data.

Should we also modify sds_data_manager/orchestration/custom_behavior/spacecraft.py? Right now I think I set it so the spacecraft jobs trigger on new repoint partitions being created. But now that we have this new partition, should we instead trigger a job based on any new pointing_attitude partitions? I'm just thinking of the scenario where a new repoint partition is added before a pointing_attitude partition, and then the spacecraft_attitude job is triggered, all before the newer pointing_attitude partitions are made.