Conversation
JSON Structure becomes the source of truth for both wire formats. Avro is a
runtime concern: the SDK compiles a document to an Avro schema on the fly, so a
developer never writes or ships an .avsc. Protobuf is a build-time concern: the
jstruct CLI emits .proto modules a service definition imports.
Two normative mapping specs under spec/, a Rust reference implementation for
each, and golden conformance corpora under test-assets/ that every future
language port is expected to pass unchanged.
Avro
- json_structure::avro::{schema_from_str, schema_from_value, compile} — a
document in, an apache_avro::Schema out.
- `any` maps to an empty record. A reader holding the compiled schema resolves
against whatever concrete record the writer put there and steps over it. At
that position the compiled schema is a reader schema, which the compiler warns
about, because writing a non-empty payload through it is rejected.
- Union defaults are placed, not merely emitted: the defaulted branch rotates to
the front of the union and the JSON Structure tag is consumed. Avro takes the
default from the first branch, so emitting it anywhere else parses and then
fails at read time.
- No logical types. Temporals, decimal, and uuid travel as string.
Protobuf
- jstruct proto, plus jstruct consolidate to resolve interdependent $import
documents into one file for either pipeline.
- Field numbers are positional and pinned by a lock file, so regeneration never
renumbers a live wire contract.
- A rootless document is valid and expected: it is a library of importable
types. This is where the Protobuf mapping deliberately diverges from Avro,
which must yield exactly one schema.
Neither compiler offers a namespace or package override. The document says where
its types live; we only translate.
CI
- rust.yml now builds and tests with --all-features. Without them the Avro
runtime and the whole CLI suite compiled to zero tests and reported success.
- protoc is installed and JSTRUCT_REQUIRE_PROTOC turns a missing toolchain from
a silent skip into a failure, so the protobuf validity check cannot quietly
stop running.
- clippy now covers --all-features --all-targets.
389 tests pass; protoc 29.3 accepts every generated file.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 033b3a57-b612-4c5a-b2d7-d9b0a6eab26d
The question that prompted this was fair and the honest answer was
uncomfortable. Avro moved actual bytes in five places, all hand-written;
the 32-case corpus only ever called `Schema::parse_str`. Protobuf moved
none at all - `protoc` proved the generated `.proto` parsed, and nothing
proved the messages could carry the data the source document describes.
That is precisely the gap a blessed golden cannot close. `expected.avsc`
proves every port agrees with the reference implementation; it cannot
prove the reference implementation is right, because it was blessed from
it. A schema that is self-consistent but wrong sails through.
So both corpora now carry hand-written instances, and both harnesses put
them on the wire.
Avro: `instance.avro.json` for all 32 valid cases, decoded, written with
`apache_avro::Writer`, read back, compared. `apache-avro` has no decoder
for the Avro JSON encoding - `Value::from(json).resolve(&schema)` matches
union branches structurally and reads `{"Circle": {...}}` as a map - so
the harness carries a small schema-driven decoder that implements the
tagging properly.
Protobuf: `instance.txtpb` for 32 of 33 valid cases, driven through
`protoc --encode` / `--decode` / `--encode` with the two binaries compared
byte for byte. No new Rust dependency; the reference implementation does
the work. `root-enum` generates no message at all, so it carries a
`no-instance.md` saying why - silence is not an acceptable way to opt out,
and the harness enforces that.
Both harnesses were mutation-tested rather than trusted for passing on the
first run: a wrong union tag, an untagged union, a bogus field, and an
all-defaults instance each fail as they should. Writing the instances also
caught one of my own misreadings - nested arrays compile straight to
`array<array<double>>` in Avro with no wrapper record, unlike proto.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 033b3a57-b612-4c5a-b2d7-d9b0a6eab26d
"Fast and deterministic" was the goal from the outset. Determinism has had a test since the corpus existed; speed had a claim and no number. Adds `examples/avro_bench.rs`, which measures against the conformance corpus itself so the benchmark cannot drift away from what is actually shipped. Three findings. The conversion costs about what parsing its own output costs - 1.16x `apache_avro::Schema::parse_str` across the corpus, ~20us for a typical schema. That is the yardstick that matters: a developer hand-maintaining an `.avsc` still pays the parse, so the conversion is roughly a doubling of a cost that was already negligible. It is linear in document size, not quadratic. 2.4-3.7us per declared property held steady from 10 properties to 10,000, where the corpus - all ~1KB documents - could never have told us. A compiler that is quadratic in property count looks fine on a conformance suite and falls over on a generated schema. The load path was doing two things for no reason, both now gone: `document.clone()` deep-copied the whole document on the overwhelmingly common path where it has no `$import` at all, now a `Cow::Borrowed`; and the compiled schema was serialized to text purely so `parse_str` could parse it straight back, when `Schema::parse` takes the JSON tree directly. Neither moved the needle much - the end-to-end path is dominated by allocation, and 145us once at startup is not worth chasing further - but both were work nobody needed done. The right answer to the cost is the one already documented: compile once and cache the `Schema`. The module docs now carry the measured figure behind that advice instead of asserting "cheap but not free" on faith. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 033b3a57-b612-4c5a-b2d7-d9b0a6eab26d
Ports the Rust reference compiler to C# as JsonStructure.Avro, so a .NET application can hand its Avro serializer a .struct.json and stop maintaining an .avsc. The seam is one line at the call site: wherever Schema.Parse took a hand-written .avsc, JsonStructureAvro.SchemaFrom takes the JSON Structure document, and everything downstream is unchanged because what comes back is an ordinary Avro.Schema. The port matches all 34 goldens byte for byte. Two things it had to get right that the Rust implementation never has to think about: JsonArray.Add<T> boxes even a string into a JsonValueCustomized<T> on .NET 8, which cannot be written without a reflection-based type-info resolver, and WriteIndented breaks lines with Environment.NewLine, which is CRLF on Windows and not what the goldens say. Both are now called out in SDK-GUIDELINES for the next port. Apache.Avro's JsonDecoder reads the Avro JSON encoding natively, including union tagging, which makes it an independent check on the hand-written decoder the Rust harness needs. It cannot be the only decoder: it throws on any schema with a self-referential type. The harness therefore decodes with its own reader, and where JsonDecoder also works asserts that both decodes serialize to identical bytes -- a check that is itself asserted, so it cannot silently degrade to nothing. Two corpus cases added, both found by mutating the compiler and watching the corpus not notice: - union-namespaced-branches: the only case whose union tag is a fullname rather than a bare type name. Without it, a port could use short names throughout and stay green. - union-default-placement: the only case where a declared default names a branch that is not already first, so the branch-rotation logic added by the adversarial review had no coverage at all. Also drops the explicit System.Text.Json package reference -- the in-box .NET 8 one is enough once the JsonValueCustomized bug is fixed, and pinning it was hiding that bug. Makes both Compile overloads return AvroCompileResult so that discarding warnings is never the shorter thing to write. Extends the .NET CI trigger to test-assets, which the new tests read. Repairs two doc comments in the Rust compiler that a bad edit had joined onto the following line. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 033b3a57-b612-4c5a-b2d7-d9b0a6eab26d
Two changes to the Avro story, both about what a schema says rather than what it encodes. A `full` mode. The compiler emitted the smallest schema that carries the data, which is right when the schema is parsed at process start and wrong when a human or a code generator reads it. `full` adds `logicalType` annotations for the temporal types and `uuid`, and appends the constraints Avro cannot express to each field's `doc`. It changes no base type and therefore no byte on the wire, and the corpus asserts that directly by compiling every case both ways and comparing encoded bytes. `compact` stays the default. Rust gets `--mode`, .NET gets `AvroOptions.Mode`. The temporal annotations use Avrotize's `rfc3339-*` names. Avro says a parser must ignore a logical type it does not recognize; `apache-avro` does, Apache.Avro throws instead, so the .NET SDK registers them with `LogicalTypeFactory` before parsing. That asymmetry is now in the spec as a requirement on ports rather than a surprise waiting for them. `decimal` moves to `bytes` with Avro's own `decimal` logical type in both modes. It was a string, which is lossless but is not what any Avro consumer expects to find. Plain JSON for the corpus instances. Avro's JSON encoding writes binary as Latin-1 code points, temporals as bare epoch numbers, and every union value wrapped in an object naming the branch. No ordinary JSON producer emits that, which makes it a poor way to describe the data a JSON Structure document is about. The instances now use the encoding specified in avrotize's avrojson.md: base64 binary, quoted `long` and `decimal`, RFC 3339 temporals, untagged unions resolved by structure. That costs the C# harness its second opinion, because Apache.Avro's `JsonDecoder` can no longer read the instances. Replaced with something stronger: `expected.avro.b64` pins the bytes each instance must encode to, so the two hand-written decoders are checked against each other rather than each against itself. Corrupting one golden fails .NET, which is the point. Blessing those bytes immediately turned up a real limit. Avro writes map entries in iteration order and `apache-avro` uses a `HashMap`, so `collections` encodes differently on two consecutive runs of the same binary. Map-bearing cases are exempt, and both harnesses assert that at least one case is exempt so the exemption cannot rot into dead code. Testing. Ten mutations against the two decoders, five each. Five survived the first pass, all of them guards the corpus cannot reach because every instance in it is meant to decode: ambiguous unions, omitted required fields, and decimals finer than their scale. Added direct negative tests for those paths in both SDKs; all ten mutations now fail. Seven earlier mutations against the full-mode compiler were already caught in both. One test was doing nothing at all. `the_two_modes_encode_identical_bytes` skipped any case whose two modes parsed equal - which, because `apache-avro` discards an unrecognized `logicalType` at parse time, was every rfc3339 case; the rest were excluded by a separate `decimal` guard. Net coverage was zero. Restructured around the two real cases: annotations the parser drops, where nothing can differ, and `uuid` and `decimal`, which become distinct schema variants and can be compared as bytes. Both counted and asserted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 033b3a57-b612-4c5a-b2d7-d9b0a6eab26d
The `avro` module's own documentation still described a single mode. Adds a Modes section with a running doctest showing `Mode::Full` producing an `rfc3339-timestamp-micros` annotation, the wire-compatibility guarantee, and the warning that the `rfc3339-*` names may need registering with a runtime that does not obey Avro's ignore-unknown-logical-type rule. The doctest is the point: it fails if the mapping ever changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 033b3a57-b612-4c5a-b2d7-d9b0a6eab26d
`full` mode followed Avrotize and appended constraints to the `doc`
string as `[minimum: 0, scale: 2]`. That is a display string: it cannot
round-trip, it collides with prose, and nothing parses it back. Avro
schemas are extensible - a parser must ignore an attribute it does not
recognize - so the constraints now travel as a `jsonStructure` attribute
beside `doc`, with their original JSON types intact.
{
"name": "total",
"type": {"type": "bytes", "logicalType": "decimal", ...},
"doc": "Order total",
"jsonStructure": {"minimum": 0}
}
A number stays a number and a pattern stays something a regex engine can
compile, at no cost to a reader that has never heard of JSON Structure.
The key set and its emission order are unchanged and still fixed; the
corpus case now scrambles the source order so that the fixed order is
actually being tested rather than coincidentally matched.
Two consequences fall out of the move, and both are specified and
tested rather than left to be discovered:
`emitDoc` no longer suppresses constraints. It governs `doc`, which is
prose for a human; constraints are metadata for a program. Coupling them
made one option mean two things. New case `full-constraints-no-doc`.
`precision` and `scale` are not repeated when Avro's own decimal logical
type already carries them. Two copies of the same fact in one schema can
only ever disagree. When a decimal falls back to a lexical string - no
precision, or a scale above it - whichever is present is annotated,
because nothing else is carrying it. New case
`full-decimal-constraints`, which covers all three shapes.
Renamed `full-doc-annotations` to `full-constraint-annotations`, since
`doc` is no longer where they live.
Ten mutations, five per SDK: drop a keyword, reorder two, emit in
compact mode, duplicate the decimal constraints, re-couple to `emitDoc`.
All ten caught.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 033b3a57-b612-4c5a-b2d7-d9b0a6eab26d
The attribute names its producer rather than its contents. It holds annotations - `minimum`, `pattern`, `maxLength` - so it is now called `annotations`, matching the section that specifies it (6.4.1, "Constraint annotations") and the corpus case that covers it. Purely a rename: the key set, the fixed emission order, the placement beside `doc`, the `emitDoc` independence, and the no-duplication rule for `decimal` are all unchanged, and no byte on the wire moves. Renaming now is free; the attribute has not shipped. Also corrects two enum doc comments that still said constraints ride on `doc`, which the previous commit made untrue. Mutating the compiler back to the old name fails two corpus tests, so the goldens pin the name rather than having drifted alongside it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 033b3a57-b612-4c5a-b2d7-d9b0a6eab26d
"Golden" is jargon that carries no meaning to a reader who has not met
it before, and the files it described are already named `expected.avsc`,
`expected-numbers.json`, and `expected-error.txt`. The prose now matches
what is on disk.
Mechanical throughout, with three renames worth noting:
every_valid_case_matches_its_golden_output
-> every_valid_case_matches_its_expected_output (Rust, both corpora)
CompilesToTheGoldenSchema
-> CompilesToTheExpectedSchema (C#)
let mut goldens
-> let mut expected_files (proto corpus)
Section headings become "Expected-output match" and the two corpus
harness headers become "Conformance-corpus harness", which says what
they are rather than what colour they are.
No behaviour changes; both suites and clippy are green.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 033b3a57-b612-4c5a-b2d7-d9b0a6eab26d
The `annotations` attribute held only value constraints, all of which land on a property. So although the compiler had emission sites on the record and enum type objects, no corpus case ever reached them: every `annotations` in the corpus sat on a field. Dead paths that two ports could silently disagree about. Units and semantic annotations are what makes those paths real, and they belong in the bucket on their own merits. A unit is a stronger case for carrying than a constraint, not a weaker one. A constraint is a rule you can re-check against the source document, and the data means the same thing either way. A unit is identity - a `double` that lost "metres" is a number that means nothing, and nothing recovers it from the bytes. Added, after the constraints and in fixed order: symbol, symbols, unit, ucumUnit, currency concepts, observedProperty, semanticRole, derivation, statistic, phenomenonTimeRelation, supportPeriod, cadence, codedValues, measurementConditioning `concepts` and `observedProperty` annotate a type, so the record and enum type objects now carry the attribute for real, pinned by `full-semantic-annotations`. Nine semantic keywords are deliberately NOT carried. They bind property names: `coordinateReferenceSystem` has a `coordinates` array naming the properties that form a coordinate, and the frame, colour, audio, spectral and temporal keywords do the same. Those are JSON Structure names, and that spec is explicit that an alternate name does not change the identity an annotation binds. Avro is the renamed world - `altnames.avro` and the name rules of section 6 mean a property can arrive under a different name. A verbatim copy would name fields that do not exist, silently, which is worse than not carrying it at all. They are dropped with a warning instead, in both modes, because unlike a constraint `full` cannot rescue them. `full-name-binding-dropped` renames its properties through `altnames` so the case shows exactly what a copy would have dangled against. Three new corpus cases, 39 -> 42. The warning's mode-independence cannot live in the corpus - a case carries one options file - so it is a unit test in both SDKs. Eight mutations, four per SDK. Seven caught; "reorder unit before symbol" survived, because no case carried both keywords on one declaration and the units group's order was therefore untested. `full-units` now does, and the mutation is caught in both SDKs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 033b3a57-b612-4c5a-b2d7-d9b0a6eab26d
The Java SDK gets the same deal Rust and .NET already have: hand it a JSON Structure document, get back an `org.apache.avro.Schema`. The `.avsc` stays an implementation detail. `JsonStructureAvro.schemaFrom` is the seam -- it goes exactly where `new Schema.Parser().parse()` used to. `AvroCompiler` sits underneath and returns the schema as a Jackson tree plus the warnings, and needs nothing but Jackson; Avro itself is an optional dependency, so nobody who only wants the validator pulls it in. Three things Java made different from the .NET port: - Jackson has a real `NullNode` where System.Text.Json has a C# `null`, so `"default": null` reads as a present default rather than an absent one. The accessor folds them together to match, and the compiler reads raw nodes wherever the difference is load-bearing. - The JSON writer is hand-rolled. Jackson's pretty printer writes `"key" : value` and indents arrays on its own scheme, and byte-exactness against the Rust reference is the whole contract. - No logical types to register. Java's Avro runtime ignores a `logicalType` it does not recognize, so the `rfc3339-*` family parses without touching Avro's static registry -- unlike Apache.Avro, which rejects it. The corpus harness ports whole: 42 valid and 10 invalid cases, seven checks, 265 assertions. It passed on the first green run, which the guidelines say to distrust, so five mutations went in to check -- a reordered annotation table, a dropped record-level `annotations`, a suppressed warning, a decimal scaled by ten, an off-by-one `long`. All five were caught. 954 tests green on JDK 21. Also extends the Java CI path filter to `test-assets/**` and `spec/**`, which the tests now depend on and the filter would otherwise skip. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 033b3a57-b612-4c5a-b2d7-d9b0a6eab26d
Register the Semantic Annotations extension in the Rust validator and map JSON Structure dates to Avro's standard date logical type across Rust, .NET, and Java. Normalize date defaults through references and unions while retaining offset-bearing temporal values as lossless strings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
Added a conformance follow-up based on validation of the Puget Sound ontology:
The rebuilt Rust |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
cargo test --manifest-path rust/Cargo.toml --all-featuresdotnet test dotnet/JsonStructure.sln --configuration Release --no-restore(1,093 passed)java/mvnw.cmd test --batch-modefromjava(954 passed)git diff --check origin/master...HEADReview notes
The commits are intentionally ordered as one cross-SDK feature. The mapping documents and shared corpora establish the contract consumed by the Rust implementation; subsequent commits refine that contract while adding .NET and Java parity.