Apache Sourcelume Sample

Tiny model, full provenance. This repo trains a char-level language model (so small it trains in seconds on a laptop CPU) and walks the complete Sourcelume loop: dataset record → training → TrainingProvenanceRecord → registry → query.

The point of the demo: the training record is born from the training run with real token counts from model/training-run.json — not authored by hand. A provenance record that says "model X was trained on dataset Y" is only interesting if you can watch it happen.

The five steps

git clone https://github.com/apache/sourcelume-sample && cd sourcelume-sample
make demo

1. Dataset record — data/provenance-record.json describes the bundled public-domain corpus (Alice's Adventures in Wonderland, 1865, Project Gutenberg ebook #11). The demo publishes this record itself: it bootstraps its own preconditions instead of assuming records it did not create.
2. Train — src/train.py trains a tiny char-level MLP (numpy only, no torch, no GPU) on the corpus and writes model/training-run.json with the honest numbers of the run.
3. Record — the trainer fills model/provenance-record.json: a TrainingProvenanceRecord referencing the dataset record by canonical record IRI, with tokens (characters per epoch) and share: 1.0 from the actual run.
4. Validate — both records are validated twice: against the spec (JSON Schema + SHACL) and against the registry's /records/validate endpoint.
5. Publish + query — with the Atlas stack up, records are published via the Python plugin; GET /records/{id} and the Atlas UI show the result.

Prerequisites

• Python 3.11+ and uv
• For the registry steps: a local Sourcelume Registry (see sourcelume-registry/dev-support, port 8082)
• For publishing: the Atlas stack (make demo-atlas)

What works today, and what deliberately does not

Step	State
Dataset record: spec + registry validation	✅ works today
Dataset record: publish to Atlas	✅ works with the Atlas stack up
Training record: spec validation	✅ works — requires a spec checkout at the model-usage-record (PR #10) state or later; set SOURCELUME_SPEC accordingly
Training record: plugin validate/publish	❌ expected to fail — the Python plugin itself only models ProvenanceRecord (its Pydantic model rejects the record client-side). The plugin needs the same type-dispatch the spec validator has (tooling follow-up; see make registry-validate).
Training record: registry validate/publish	❌ expected to fail until the registry learns the TrainingProvenanceRecord type (registry follow-up to PR #10). Direct curl -X POST .../records/validate shows the registry's own rejection (HTTP 422, must be the constant value 'ProvenanceRecord') — the demo's exhibit for that follow-up.

This demo builds on PR #10 as given: the training record type, the type-dispatching validator and the record examples come from that branch.

Honesty notes

• The demo record is unsigned (Sourcelume Attest is still a discussion).
• It is a producer's own claim — no certification, no adjudication, no "approved" flag anywhere.
• disclosure: complete is only honest because the demo knows its data. Realistic partial/undisclosed cases are covered by the spec examples.
• One token = one character (char-level model); tokens are per epoch, following the Mímir Table-10 convention.