Skip to content

feat: open CRS, Z/M geometry, COG raster load, array attr metadata, graph AS OF, Calvin key placement - #440

Merged
farhan-syah merged 12 commits into
mainfrom
fix/mysa-eo-gaps
Oct 25, 2026
Merged

farhan-syah merged 12 commits into
mainfrom
fix/mysa-eo-gaps

Conversation

@farhan-syah

Copy link
Copy Markdown
Member

Summary

NodeDB reads Earth-observation data in any coordinate reference system. It loads Landsat and Sentinel COGs over HTTP and S3, stores 3D and measured geometry, carries band metadata, and runs graph reads AS OF a past system time. Calvin transactions with large edge batches lock only the keys each participant owns.

What changed

Coordinate reference systems

Area Change Paths
Open SRID Srid(NonZeroU32) replaces the closed enum. A code unknown to the registry is refused at every external boundary. nodedb-srid/, nodedb-types/src/geometry/
CRS registry Pure-Rust crate, wasm32. EPSG table v12.029, 21 projection methods, Helmert 3/7-parameter datum shifts through ECEF. Parsers and writers for PROJ, PROJJSON and WKT2. nodedb-crs/
GDM2020 EPSG 12287 and 25314–25322, with time-dependent Helmert. ST_Transform(g, srid, epoch). nodedb-crs/data/
User CRS CREATE SPATIAL_REF_SYS / DROP SPATIAL_REF_SYS, the nodedb.spatial_ref_sys view, Raft-replicated. DROP of a code in use is refused. nodedb/src/control/
Measures Planar measures honour the CRS unit. Raster GeoKeys match the registry by datum. nodedb-spatial/, nodedb-raster/src/crs.rs

Geometry

Area Change
Z/M POINT Z/M/ZM and EMPTY in WKT, WKB, EWKB and GeoJSON. Columnar and strict collections store EWKB.
Errors A WKT or WKB parse error is a statement error, never a NULL.
Geodesic Measures run on the CRS ellipsoid through geographiclib-rs.
Overlay ST_Union, ST_Intersection, ST_Difference, ST_Buffer and others run through geo BooleanOps and Buffer.
Functions About 40 new ST_* functions, including ST_3DDistance, ST_3DDWithin, ST_MakeValid, ST_Relate and ST_LocateAlong.
Search box The spatial refine step no longer undercovers its search box. It sizes the box by the CRS metric.

Raster load

Area Change Paths
Reader Sans-IO TIFF and BigTIFF reader with overviews, masks and GDAL_METADATA. Replaces the tiff crate. nodedb-raster/src/tiff/
Codecs LZW, deflate, PackBits, ZSTD, JPEG, LERC (with deflate and ZSTD wrappers), predictors 2 and 3. Every decode allocation is capped (MAX_DECODE_CAPACITY). nodedb-raster/src/codec/
Load planning Band mapping, grid alignment and nodata, shared with Lite. nodedb-raster-load/
Sources COPY ... FROM takes several files, one per band, plus HTTP and S3 range fetches pinned by ETag. nodedb/src/control/
Options OVERVIEW, WINDOW, ADOPT_METADATA, per-band masks.
Egress One [egress] policy serves raster fetches, JWKS, webhooks and traces. HTTPS only, private CIDRs refused unless allowlisted, redirects checked again. nodedb/src/control/security/
Credentials CREATE CREDENTIAL, sealed by the [credential_encryption] key. The statement text is redacted from Raft and the audit log.

Array attributes

  • AttrSpec carries nodata, scale, offset and units.
  • New INT8 attribute type.
  • New reducers: median, percentile, stddev, mode, histogram.

Bitemporal and graph

  • Graph traversal, MATCH and the graph algorithms take AS OF SYSTEM TIME and AS OF VALID TIME. A per-core bounded historical CSR cache serves them.
  • GRAPH INSERT EDGE ... VALID_TIME <from> [UNTIL <until>].
  • Document, columnar and edge versions use one ordinal clock. The leader stamps each replicated write, and every replica installs the same stamp.
  • Each statement pins one AS OF cut. Every serving node waits for that cut's marker before it reads.
  • A columnar UPDATE or DELETE on a bitemporal collection keeps the history.

Cluster

  • A vShard that leaves a node purges its state from every engine on that node, CRDT included.
  • The CRDT peer id includes the collection incarnation. Sync refuses a delta for a replaced incarnation.

Calvin key placement

  • HomedPair scopes each graph lock pair to one vShard. Each participant locks only the pairs it owns.
  • PartTasks wire: each target decodes only its own tasks. A homes list that differs from routing is refused.
  • ReplayCache: each node decodes a committed entry once and shares it across its catch-up replays. The cache has a 64 MiB budget.
  • SortedVec decode refuses input that is not strictly ascending.
  • Lock manager: acquire returns the transactions it promoted. Gate watchers wake on release.

Licensing

  • New CLA.md. CONTRIBUTING.md and the PR template point to it.
  • The BUSL-1.1 licensor is Biz Impulse Solutions Sdn. Bhd.
  • Each Apache-2.0 library crate has a NOTICE file.

Why

Symptom Cause
Malaysian datasets in GDM2020 or Kertau/Timbalai RSO could not load. The SRID was a closed enum of a few codes.
Landsat COGs (one file per band, ZSTD or LERC, over HTTP) could not load. The tiff crate had no LERC and no raw tile access. COPY took one local file.
3D survey geometry lost its Z. Geometry carried XY only.
Graph history could not be read. Graph reads ignored AS OF.
A Calvin transaction with a large edge batch hit DeadlineExceeded. Every participant received and expanded every edge key: O(P×K) lock work. Replay decoded the header once per vShard, 1026 times. Fixed, the work fell from 68.4 G to 6.2 G instructions.

Behaviour changes

  • An SRID unknown to the registry is refused at every external boundary.
  • A WKT or WKB parse error fails the statement. Before, it returned NULL.
  • Geometry is no longer #[non_exhaustive].
  • The sync wire version changes. DeltaPushMsg carries collection_incarnation, so old clients are refused at handshake.
  • The SortedVec and TxClass msgpack encodings change. Pre-1.0, so there is no compat shim.

Known limits

  • A write outside Calvin, on a node whose clock leads, can sit above a "now" cut by up to the clock skew. docs/bitemporal.md states this.
  • Timeseries _ts_system stays an ingest stamp.

Lite

Lite needs these changes when it moves to 0.7:

  • SystemTimeScope::Pinned { ordinal, cut_marker } arms in the array, columnar, timeseries and graph adapters.
  • SpatialPredicate::DWithin3D, the Result signatures of the WKT/WKB readers, and SridGeometry msgpack [srid, geometry, z, m].
  • DeltaPushMsg.collection_incarnation.
  • system_from / erase_history on the ColumnarOp update, delete and truncate variants.

Not in this PR

Dense tiles, NetCDF/HDF/Zarr/FITS/STAC readers, grid operators, ARRAY_WINDOW and Arrow results, COG/GeoParquet export, and OGC services.

How to check it

Command Result
cargo nextest run --workspace --exclude nodedb-cluster-tests --all-features --no-fail-fast --retries 0 23622 passed, 3 skipped
cargo nextest run -p nodedb-cluster-tests --all-features --no-fail-fast --retries 0 531 passed
cargo clippy --workspace --all-targets --all-features --profile ci -- -D warnings clean
scripts/ci/check_* static gates, cargo deny check, wasm32 check clean

Add CLA.md and point CONTRIBUTING.md and the pull request template at
it. Change the BUSL-1.1 licensor and copyright line to Biz Impulse
Solutions Sdn. Bhd. Add a NOTICE file to each Apache-2.0 library crate.
Add a NOTICE file to nodedb-raster, matching the other Apache-2.0
library crates.
…ata, and bitemporal graph reads

Spatial:
- Add nodedb-crs and nodedb-srid crates for coordinate reference
  system definitions and SRID resolution, with a catalog-backed
  spatial_ref_sys table and DDL.
- Rework nodedb-spatial around typed errors, geodesic and 3D
  measures, WKT/WKB readers with Z/M ordinates, and overlay ops.
- Extend the raster crate with TIFF/COG decoding, codecs, and
  sources, and add nodedb-raster-load for bulk array load.

Array:
- Add attribute metadata, reducers, and a physical query layer.

Bitemporal and graph:
- Add statement-level system/valid-time cuts, columnar system
  stamps, persisted node labels, and as-of graph reads.

Cluster and security:
- Purge state on vShard departure, add credential DDL, a JWKS
  registry, egress policy, and a WAL key vault.
Sequencer:
- Cache committed entries decoded once so every armed vShard's
  catch-up replay reuses them, bounded by a byte budget and pruned as
  catch-ups clear or drop.
- Reject out-of-order or repeated items when decoding a sorted set
  or a class's body_plans, with matching error variants.
- Add HomedPair to scope a graph lock pair to one vShard or to every
  participant.
- Carry whole-task parts as a decoded part tasks form.

Scheduler and planner:
- Rework the lock manager's acquire, release, and wound-wait paths.
- Assemble part tasks in the driver and prepare submissions in the
  planner before routing.
- Extend write admission waiting and read-set handling to match.
Comment thread nodedb-crs/src/parse/helmert_params.rs Dismissed
Comment thread nodedb-crs/src/parse/helmert_params.rs Dismissed
Comment thread nodedb-crs/src/parse/proj_string.rs Dismissed
Comment thread nodedb-crs/src/parse/projjson_writer.rs Dismissed
Comment thread nodedb-crs/src/parse/projjson_writer.rs Dismissed
Comment thread nodedb-sql/src/ddl_ast/parse/credential.rs Dismissed
Comment thread nodedb-sql/src/ddl_ast/parse/credential.rs Dismissed
Comment thread nodedb/src/control/security/keystore/wal_key.rs Dismissed
Comment thread nodedb/src/control/server/shared/session/audit_context.rs Dismissed
Comment thread nodedb/src/control/server/shared/session/audit_context.rs Dismissed
Add libFuzzer targets for CRS definition parsing and spatial geometry
parsing, with seed corpora and dictionaries, and register them in the
fuzz workflow.
Run the EPSG table generator in check mode in the static gates and test
workflows. Suppress cleartext-logging alerts on test-only assert
messages in CRS and SQL parser tests.
PROJ runs on the host libm, so the last digits of generated fixture
coordinates differ between glibc versions. The fixture check now
requires identical lines and fields, and accepts coordinate columns
that agree within a small relative tolerance.
@farhan-syah farhan-syah added the run-ci Opt this PR into the full test suite; re-add to force a re-run label Oct 24, 2026
Fixture coordinate error scales with the Earth-sized intermediates in
the projection math, not with the coordinate itself, so a relative
tolerance rejects valid output for local grids near their origin.
Compare projected coordinates in metres and geographic coordinates in
degrees, each against its own absolute tolerance, to absorb differences
between glibc versions and between x86_64 and aarch64.
Adapt JWKS ES256/ES384 verification to the new API by parsing SEC1
points with VerifyingKey::from_sec1_bytes. Add sign/verify round-trip
tests for both curves and a test that an off-curve point is refused.

Drop the RUSTSEC-2021-0153 advisory ignore from deny.toml, since the
dependency update removes the crate that triggered it.
@farhan-syah
farhan-syah merged commit 4acccf1 into main Oct 25, 2026
17 checks passed
@farhan-syah
farhan-syah deleted the fix/mysa-eo-gaps branch October 25, 2026 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci Opt this PR into the full test suite; re-add to force a re-run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants