Repository navigation
feat: open CRS, Z/M geometry, COG raster load, array attr metadata, graph AS OF, Calvin key placement - #440
Merged
Merged
Conversation
Add CLA.md and point CONTRIBUTING.md and the pull request template at it. Change the BUSL-1.1 licensor and copyright line to Biz Impulse Solutions Sdn. Bhd. Add a NOTICE file to each Apache-2.0 library crate.
Add a NOTICE file to nodedb-raster, matching the other Apache-2.0 library crates.
…oad, and bitemporal graph reads
…ata, and bitemporal graph reads Spatial: - Add nodedb-crs and nodedb-srid crates for coordinate reference system definitions and SRID resolution, with a catalog-backed spatial_ref_sys table and DDL. - Rework nodedb-spatial around typed errors, geodesic and 3D measures, WKT/WKB readers with Z/M ordinates, and overlay ops. - Extend the raster crate with TIFF/COG decoding, codecs, and sources, and add nodedb-raster-load for bulk array load. Array: - Add attribute metadata, reducers, and a physical query layer. Bitemporal and graph: - Add statement-level system/valid-time cuts, columnar system stamps, persisted node labels, and as-of graph reads. Cluster and security: - Purge state on vShard departure, add credential DDL, a JWKS registry, egress policy, and a WAL key vault.
Sequencer: - Cache committed entries decoded once so every armed vShard's catch-up replay reuses them, bounded by a byte budget and pruned as catch-ups clear or drop. - Reject out-of-order or repeated items when decoding a sorted set or a class's body_plans, with matching error variants. - Add HomedPair to scope a graph lock pair to one vShard or to every participant. - Carry whole-task parts as a decoded part tasks form. Scheduler and planner: - Rework the lock manager's acquire, release, and wound-wait paths. - Assemble part tasks in the driver and prepare submissions in the planner before routing. - Extend write admission waiting and read-set handling to match.
Add libFuzzer targets for CRS definition parsing and spatial geometry parsing, with seed corpora and dictionaries, and register them in the fuzz workflow.
Run the EPSG table generator in check mode in the static gates and test workflows. Suppress cleartext-logging alerts on test-only assert messages in CRS and SQL parser tests.
PROJ runs on the host libm, so the last digits of generated fixture coordinates differ between glibc versions. The fixture check now requires identical lines and fields, and accepts coordinate columns that agree within a small relative tolerance.
Fixture coordinate error scales with the Earth-sized intermediates in the projection math, not with the coordinate itself, so a relative tolerance rejects valid output for local grids near their origin. Compare projected coordinates in metres and geographic coordinates in degrees, each against its own absolute tolerance, to absorb differences between glibc versions and between x86_64 and aarch64.
Adapt JWKS ES256/ES384 verification to the new API by parsing SEC1 points with VerifyingKey::from_sec1_bytes. Add sign/verify round-trip tests for both curves and a test that an off-curve point is refused. Drop the RUSTSEC-2021-0153 advisory ignore from deny.toml, since the dependency update removes the crate that triggered it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NodeDB reads Earth-observation data in any coordinate reference system. It loads Landsat and Sentinel COGs over HTTP and S3, stores 3D and measured geometry, carries band metadata, and runs graph reads
AS OFa past system time. Calvin transactions with large edge batches lock only the keys each participant owns.What changed
Coordinate reference systems
Srid(NonZeroU32)replaces the closed enum. A code unknown to the registry is refused at every external boundary.nodedb-srid/,nodedb-types/src/geometry/nodedb-crs/ST_Transform(g, srid, epoch).nodedb-crs/data/CREATE SPATIAL_REF_SYS/DROP SPATIAL_REF_SYS, thenodedb.spatial_ref_sysview, Raft-replicated.DROPof a code in use is refused.nodedb/src/control/nodedb-spatial/,nodedb-raster/src/crs.rsGeometry
POINT Z/M/ZMand EMPTY in WKT, WKB, EWKB and GeoJSON. Columnar and strict collections store EWKB.NULL.geographiclib-rs.ST_Union,ST_Intersection,ST_Difference,ST_Bufferand others run throughgeoBooleanOps and Buffer.ST_*functions, includingST_3DDistance,ST_3DDWithin,ST_MakeValid,ST_RelateandST_LocateAlong.Raster load
GDAL_METADATA. Replaces thetiffcrate.nodedb-raster/src/tiff/MAX_DECODE_CAPACITY).nodedb-raster/src/codec/nodedb-raster-load/COPY ... FROMtakes several files, one per band, plus HTTP and S3 range fetches pinned by ETag.nodedb/src/control/OVERVIEW,WINDOW,ADOPT_METADATA, per-band masks.[egress]policy serves raster fetches, JWKS, webhooks and traces. HTTPS only, private CIDRs refused unless allowlisted, redirects checked again.nodedb/src/control/security/CREATE CREDENTIAL, sealed by the[credential_encryption]key. The statement text is redacted from Raft and the audit log.Array attributes
AttrSpeccarriesnodata,scale,offsetandunits.INT8attribute type.Bitemporal and graph
MATCHand the graph algorithms takeAS OF SYSTEM TIMEandAS OF VALID TIME. A per-core bounded historical CSR cache serves them.GRAPH INSERT EDGE ... VALID_TIME <from> [UNTIL <until>].AS OFcut. Every serving node waits for that cut's marker before it reads.UPDATEorDELETEon a bitemporal collection keeps the history.Cluster
Calvin key placement
HomedPairscopes each graph lock pair to one vShard. Each participant locks only the pairs it owns.PartTaskswire: each target decodes only its own tasks. A homes list that differs from routing is refused.ReplayCache: each node decodes a committed entry once and shares it across its catch-up replays. The cache has a 64 MiB budget.SortedVecdecode refuses input that is not strictly ascending.Licensing
CLA.md.CONTRIBUTING.mdand the PR template point to it.NOTICEfile.Why
tiffcrate had no LERC and no raw tile access.COPYtook one local file.AS OF.DeadlineExceeded.Behaviour changes
NULL.Geometryis no longer#[non_exhaustive].DeltaPushMsgcarriescollection_incarnation, so old clients are refused at handshake.SortedVecandTxClassmsgpack encodings change. Pre-1.0, so there is no compat shim.Known limits
docs/bitemporal.mdstates this._ts_systemstays an ingest stamp.Lite
Lite needs these changes when it moves to 0.7:
SystemTimeScope::Pinned { ordinal, cut_marker }arms in the array, columnar, timeseries and graph adapters.SpatialPredicate::DWithin3D, theResultsignatures of the WKT/WKB readers, andSridGeometrymsgpack[srid, geometry, z, m].DeltaPushMsg.collection_incarnation.system_from/erase_historyon theColumnarOpupdate, delete and truncate variants.Not in this PR
Dense tiles, NetCDF/HDF/Zarr/FITS/STAC readers, grid operators,
ARRAY_WINDOWand Arrow results, COG/GeoParquet export, and OGC services.How to check it
cargo nextest run --workspace --exclude nodedb-cluster-tests --all-features --no-fail-fast --retries 0cargo nextest run -p nodedb-cluster-tests --all-features --no-fail-fast --retries 0cargo clippy --workspace --all-targets --all-features --profile ci -- -D warningsscripts/ci/check_*static gates,cargo deny check, wasm32 check