Composite vars, now it's das3 - #23
Merged
Merged
Conversation
The initial work on an auto CDF reader prompted questions on the nature of the das3-basic-stream schema. In order to not shoe-horn in concepts to a limited stream vocabulary this feature set re-structures DasVar by adding a successor DasSet which uses a new syntax. The goal for these changes is to manage less, but more composable code and handle the new real-world snippets included in the feature set. Next steps are to update the test examples, and write the set.c and generator.c bodies. To trigger a compile of the new code, use -DDAS_FUTURE. Written-by: Claude Opus 4.8
In preparation for actual C development the test cases have been updated to the newest schemas. Written-by: Claude Fable 5
* Generators: generator.c gives the array, sequence, and constant value
sources;
* set.c gives the formalism table, the ordered binop rule registry,
and scalar sets.
* Linear is an explicit formalism row so ALL arithmetic dispatches
through the registry (misses refuse)
* Structural family (scalar/string/blob/composite) is stored, rich
presentation is derived from structure + formalism.
* New unit test for sets initiated
Next: migrate the DasVar consumer TUs to DasSet/DasGen, then teach
dataset_hdr3.c the new wire elements.
Written-by: Claude Fable 5
* The new stream format now decodes, vectors as a top level element are gone, replaced by 'composite' elements with a 'vector' formalism * Extension codec trigger attribute renamed to 'embedded' * native-byte encoding renamed from 'blob' to 'raw' * The new Set classes are: 'DasSet' for no internal structure and 'DasIntrSet' for variables with internal structure. * Computations now dispatch through the das_form_findRule() function * Dataset groups (a das2 concept) now work for das3 streams * All variable.h (DasVar) and associated items are out of the build Next up: das3_cdf and das3_spice updates, then the mechanical rename back to DasVar. Written-by: Claude Fable 5
The parallel constructor DasAry_subSetIn wasn't initializing all needed fields, now it does. Written-by: Claude Opus 5
The subset in functions of the old DasVar classes were ported and bugs of opportunity were fixed. New test/TestDim.c added. Written-by: Claude Opus 5
* Schema refactor: The <formalism type="X"><X .../></formalism> pair collapses to one flat <ops kind="X" .../>. This allows unknown mathematical objects to pass through das2C unhindered, though no automatic calculations can be performed on them. * External plus internal indices are checked against DASIDX_MAX * A test of composite variables with multiple internal indices added. Written-by: Claude Opus 5
* ops refs= are removed from `<ops>` who didn't know what to do with them anyway. * opts body= folded into the context on discovery * More defaults are dropped on write: use="center", units="" Written-by: Claude Opus 5
* The great macro shuffle. Array index macros in array.h, all degenerate index and other index concept macros in generator.h. * Move generator index string parsing to generator.h/c from the uber file: dataset_hdr3.c Note: This breaks das2py and das2dlm, will fix later. Written-by: Claude Opus 5
Written-by: Claude Opus 5
The big one: The XML elements <scalar>, <bytes>, & <composite> are all now represented by a single das2C DasSet derived class. Written-by: Claude Opus 5
Governing invariant, stated in form.h and set.h: Forms compute, Sets walk. Formalisms provide encode/decode functionality for thier attributes and more importantly, callbacks for binary operations which are triggerd by creation of a DasBinSet (later DasVarBin) Other cleanup includes making the context table a full object and small error value return macros. Next up: geovec get's a divorce and becames geoloc and vector. Written-by: Claude Opus 5
* Schema change: <ops kind="geovec"> is replaced by two formalisms. kind="vector" frame-tagged free vector; surface= and center= illegal kind="geoloc" body-centered position; center= required * An old stream saying "geovec" reads as DasFormGeneric: carried, math refused. Coordinate with dastelem. * New: CTX_BODY and the <body> context element. Component-system conversion for cart/cyl/sph/centric, using cspice when linked. Partial vectors add slot-wise on the union of directions present. * Gone: das_formalism and the binop rule registry (set.c), DasSet_vecMap, das_makeCompLabels, the six DasStream_*Ctx forwards, form_geovec.*. * DOES NOT LINK. das_value_binop, das_datum_box, DasSet_itemAt and DasGen_elemShape are declared and empty Written-by: Claude Opus 5
Formalisms that have extra metadata carry it internally there is no stream reference item list anymore the cost of using it was too high, maybe it will come in some other form. Note: Still dosen't build, next comes the renames Written-by: Claude Opus 5
Over the last many years more and more functionality has been added to the library to support das3 idioms. As of this commit a breaking change was finally pushed. The header directory became "das3" instead of "das2". The temporary DasSet name has been moved to DasVar Written-by: Claude Opus 5
* vtGeoVec nolonger present, everything has move to the new extensible formalism architecture * The relation ship between composite types and das_datum is finally settled. * das_value_binop bug fixes and overflow checks corrected * bug fixes of opportunity handled. * New unittest programs Written-by: Claude Opus 5
* The current DasVar function call interface should handle
all cases of retrieving values, up to including
ragged runs of externally bounded vector sequences.
* Reference counting has one rule, nothing steals. Anything
that adds to the reference count needs the caller to
decrement the count manually.
* New tests were addedfir TestGen and TestForm
* Example streams were partially split out from the test
area to 'examples'
Next up: Finishing the example move
Insuring binary operations over ragged runs and
values fetch data properly
Written-by: Claude Opus 5
All example files now read as expected except for ex22 which indicates a bug in how das3_text emits vector component order Examples worth using by a downstream stream author are all in ./examples bug fix: The header <values> reader no longer hands DasCodec_decode the length of the incoming expat chunk as the size of its 64 byte underflow buffer. bug fix: Consuming non-whitespace text without decoding a value is now an error where it happens Written-by: Claude Opus 5
* The previous "settled" DasVar API conflated two different operations: 1) Squaring off variable data via fill values and 2) materalizing generator output. Those concepts are now denoted by explicit function names for the caller * Implimentation now understands that Variables are Dataset shaped, recovering past behavior * Complex numbers have a formalism * New unit tests for DasVar stressing subset operations * The following bugs were fixed: - das3_text described a 2-component vector as 3-component - _geoloc_encode wrote body= twice - DasForm_isVector, DasForm_isGeoLoc NULL form dereference - _DasGenAry_subsetView called every ragged index out of range Written-by: Claude Opus 5
* das3_spice compiles again: locations are geoloc formalisms. Still
untested end to end, test kernels are the next job.
* Collapsed the number of formalism specific function calls using
DasForm_isKind(pForm, DAS_FORM_VEC) and DasVar_formIs(). The
specific formalisim is play is treated more as data than structure.
* das_datum_toDbl() returns bool with an out parameter, the new
das_datum_toDoubles() handles vtComposite datums.
* Updated default geoloc symplos to follow the common coventions
in geodesy: detic lambda, phi, h; centric r, lambda,
phi; The Graphic component system is west-positive and ordered (lat,
lon, height) to stay right handed; elsewhere das3_spice emits detic.
* Bug Fixs: ellipsoidal flattening was always zero; graphic longitude
used a body list naming the Earth-Moon barycenter instead of Earth;
DasDim_shape() merged an uninitialized buffer; uri.c cast a fill
value to int64_t; das_datum_toDbl() never handled vtByte.
DasVar_shape() pre-fills it output buffer.
Written-by: Claude Opus 5
* DasForms are owned objects not a share refcount resource * DasForm construction is finalized by a validate() call. * DasVar_compSym() answers for any composite in storage order * DasVar_compLabels() is provided as default component labeler * Rotation representations "matrix|quaternion" implemented and component order may be set with sysorder= * Unlike vectors, rotations do not provide defaults for absent components. * Bug fix: das3_csv now gives every composite a column per component, not just vectors. * Bug fix: _DasProp_next now terminates every element it copies out. Written-by: Claude Opus 5 Written-by: Claude Fable 5.1
* No code changes at all in this commit Written-by: Claude Fable 5.1
* This restores the symmetry between DasVar::subset() and DasVar::get() * DasVar_valType() answers vtComposite for any variable with an internal shape. * Bug fix: the hex rendering of a byte-sequence datum printed only its first byte, and single digits for bytes below 0x10. * Bug fix: ex17 kind="geovec" --> "vector" Written-by: Claude Fable 5.1
Bugs in das3_csv have been addressed and in the process of determining how to handle ragged datasets it was determined that flattening everything to Rank-1 would provide the most sensable output for people that are likely to use CSV data. New CSV regression tests have been added. An option to *not* combine reference + offset variables has been added Written-by: Claude Fable 5.1
* das3_cdf writes every composite, even generic ones with only pass-through support. * Only one units value is supported on a composite, but known composite values my have components that are understood to be in units of degrees. Radians aren't supported on the wire in v3.0. * ISTP proposed attributes: COORDINATE_SYSTEM, TENSOR_ORDER, REPRESENTATION_1, FRAME_ORIGIN are supported by das3_cdf * Mission-specific attribute renames are supported in das3_cdf via command the command line * New Tests: ex43 (complex pairs, both systems, over a sequence coordinate) and Tracers MSC/MAG cut-downs under test/cdfs. * Bug Fix: Makefile test target now keys off of SPICE= and CDF= environment variables. Developed-with: Claude Fable 5.1
* composite variables now print thier properties in a uniform manner, though this is different from the pre-3.0 format. * A new program, das3_info replaces das3_text and makes it more useful. The old capability of just testing to see if a stream is readable is retained. * Bug fix: During conversion to vtComposite handling variable info printing lost much of it's functionality. This patch restores it. * Bug fix: DasBug_initReadWrite never actually wrapped it's given external array, now it does. * Bug fix: Time sequences based on a reference time structure and an offset sequence (in the stream index!) are now supported. * das_vt_fromStr now accepts 'struct' for a vtTime output though this choice may prove to be be fragile in future stream versions * New info printing regression tests added: test/das3_info_test.sh Developed-with: Claude Fable 5.1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a massive re-think of how collections of values that follow mathematical rules such as vectors, rotations, etc. should be represented on a stream. It finally brings in band concepts that have bedeviled das2 for a decade such as "when is an addition rule not defined between two values" etc. The variable class has been split such that it's a composition of three objects:
Also all aspects of ragged data handling are implemented, and except for the item in objects, everything in the schemas is supported by the C library.