Port ripple benchmark pipeline from Nomad to AWS Batch - #2
Open
robgpita wants to merge 2 commits into
Open
Conversation
…C cataloging Ports the ripple1d/flows2fim benchmark extent pipeline from the old Nomad setup (flows2fim-runner) to AWS Batch, following the existing GFM/GFM_EXP split/worker/merge pattern with per-library job granularity. - ingest/ripple/extent_worker.py: Batch worker, structural port of the original Nomad F2FWorker; now writes COG directly (was VRT) and accepts non-numeric library ids (ohio_rfc, mn_Other, nc_Other) - ingest/ripple/batch_split.py, batch_merge.py, batch_utils.py: manifest discovery, idempotent skip logic, run summary/reconciliation - ingest/ripple/cog_worker.py: new Batch phase to convert existing VRT-as-.tif output to real GeoTIFF COGs in place, for libraries generated before the output_format switch - ingest/ripple/ripple_col.py: STAC cataloging updated for v0.11.x — wires up ble/mip/mn/nc, skips libraries with no raster output yet instead of crashing, bootstraps the catalog if missing - ingest/ripple/worker_config.yaml: S3 paths, naming templates, flow intervals folded in as single source of truth; output_format now cog - Dockerfile.ripple: GDAL + flows2fim image for the ripple worker - terraform/: ripple ECR repo, 4 job definitions (split/worker/merge/cog), compute resource sizing from observed usage, desired_vcpus safety fix - scripts/run_pipeline_prefect.py: ripple pipeline support, --items-per-job, --limit, --cog-only, --run-cog-conversion flags - scripts/build_and_push.sh, batch-entrypoint.sh: ripple image build support Verified in production: full raster run (1,120/1,238 libraries succeeded, 118 known flows2fim data failures), full COG conversion (1,234/1,238 converted, byte-level spot-checked), STAC cataloging tested end-to-end against real S3 data.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports the ripple1d/flows2fim benchmark extent pipeline from Nomad
(
fim-misc/flows2fim-runner) to AWS Batch, following the existing GFM/GFM_EXPsplit/worker/merge pattern with per-library job granularity, and adds STAC
cataloging for the resulting
ripple-fim-collection.ingest/ripple/extent_worker.py,batch_split.py,batch_merge.py,batch_utils.py— Batch worker + manifest discovery/idempotent skiplogic, structural port of the original Nomad worker. Writes COG directly
(was VRT); accepts non-numeric library ids (
ohio_rfc,mn_Other,nc_Other).ingest/ripple/cog_worker.py— one-time Batch phase to convertVRT-as-
.tifoutput (from before theoutput_formatfix) to realGeoTIFF COGs in place.
ingest/ripple/ripple_col.py— STAC cataloging updated for v0.11.x:wires up
ble/mip/mn/nc, skips libraries with no raster outputinstead of crashing, and now writes a
skipped_libraries_<timestamp>.txtreference to S3 so exclusions are auditable.
Dockerfile.ripple,terraform/,scripts/run_pipeline_prefect.py—ripple ECR repo + 4 job definitions,
--items-per-job/--limit/--cog-only/--run-cog-conversionflags, compute sizing from observedusage.
README.md,docs/aws-batch-pipeline.md— ripple folded into the sharedrunbook alongside GFM/GFM_EXP (previously undocumented at the top level).
Testing
ripple_v0.11.xlibrary set, twice (once for initial NC/MN backfill, once after further
NC/MN data landed) — both completed successfully.
1,251 items cataloged; verified collection.json + a sample of
items across all 4 sources resolve to real, non-empty S3 assets with
correct
ripple_v0.11.xpaths; verified 0 unexpected/extra librarydirectories and structural completeness (thumbnail + model_domain + all
6 flow-interval extents) for 1,245/1,251 libraries, with the remaining 6
matching a known, already-documented partial-data failure mode.
libraries on a subsequent STAC run. Only new/changed libraries get
reprocessed, not a full re-run.
before being overwritten by the real run.