diff --git a/.gitignore b/.gitignore index 666715b..6d36814 100644 --- a/.gitignore +++ b/.gitignore @@ -26,6 +26,14 @@ demo/**/target/ demo/**/logs/ demo/**/dbt_packages/ +# demo recorder toolchain (tapes/pages are committed; the recorder's deps + rendered +# media are regenerable — `demos/standalone/build.mjs` + `record.mjs` rebuild them). +# The committed copies live where they're used: README GIFs in docs/assets/demos/, +# landing videos in demos/landing/media/. +demos/standalone/node_modules/ +demos/standalone/package-lock.json +demos/standalone/out/ + # duckdb build artifacts *.duckdb *.duckdb.wal @@ -39,3 +47,6 @@ demo/**/dbt_packages/ # Vendored erwin XSD schemas — reference only, must NOT ship in a public repo. erwin/ + +# Modelith CLI demo renders (regenerable via demos/render.sh) +demos/out/ diff --git a/README.md b/README.md index 6f88dc1..6e3da00 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,10 @@ owns state. [![license](https://img.shields.io/badge/license-Apache--2.0-lightgrey)](#license) [![code style: ruff](https://img.shields.io/badge/lint-ruff-black)](https://docs.astral.sh/ruff/) +![Design a model in git, generate contract-enforced dbt, and compile to a data contract — the mdl CLI beside its live ER diagram](docs/assets/demos/Main.gif) + +Design in git → generate dbt → compile to a contract. [See all six flows ↓](#video-demos) + ![Modelith canvas: a seven-entity pension IBoR model with crow's-foot relationships](docs/assets/canvas.png) ## Try it in 2 minutes @@ -42,6 +46,46 @@ term — the demo starts a local ontology server for you. See [Ontology and knowledge graph, step by step](#ontology-and-knowledge-graph-step-by-step) for the ontology workflow. +## Video demos + +Six flows, end to end. Every command and every line of output is real — the `mdl` CLI on the +left, the live ER diagram on the right. + +### Design-first — a blank repo to running dbt +Scaffold a model, add entities (ULIDs minted for you), validate, and compile to +contract-enforced dbt plus an ODCS data contract. + +![Design-first: mdl init, new entity, validate, generate, export contract, with the ER diagram building alongside](docs/assets/demos/Main.gif) + +### Adopt a warehouse — reverse a legacy dbt project +Point it at a warehouse with no data model: staging excluded, SCD2 and Data Vault detected, +surrogate keys stripped, every inference in a reviewable ledger. + +![Adopt a warehouse: mdl reverse detecting SCD2, Data Vault, and stripping surrogate keys, with raw SQL becoming clean ER cards](docs/assets/demos/Adopt.gif) + +### The drift gate — model and warehouse in lockstep +A column lands in the warehouse; drift classifies it (additive, not breaking) and reconcile +folds the safe change into the model. Nothing is ever lost. + +![The drift gate: no drift, then an additive column caught by mdl drift --check and folded into the model by --reconcile](docs/assets/demos/Drift.gif) + +### Ontology anchoring — align to an industry vocabulary +Search FIBO, align an entity to a standard term, and track coverage — reproducible, offline. + +![Ontology anchoring: mdl ontology search returning fibo:PartyInRole, aligning counterparty, and 100% coverage](docs/assets/demos/Ontology.gif) + +### Erwin import — bring an erwin model into git +Import an erwin XML export straight into clean logical YAML you can version, diff, and +generate from. + +![Erwin import: mdl import erwin turning an erwin XML export into clean Modelith ER cards](docs/assets/demos/Erwin.gif) + +### One model, many targets — a contract factory +From one definition: contract-enforced dbt, an ODCS contract, Pydantic, SQL DDL, Mermaid, +DBML, RDF/SHACL, and a Neo4j graph schema — each generated deterministically. + +![One model, many targets: the model fanning out to dbt, ODCS, Pydantic, SQL, Mermaid, and DBML](docs/assets/demos/Compile.gif) + ## Why Warehouse teams keep their meaning in three disconnected places: an ER tool that diff --git a/demos/DESIGN-BRIEF.md b/demos/DESIGN-BRIEF.md new file mode 100644 index 0000000..7a01734 --- /dev/null +++ b/demos/DESIGN-BRIEF.md @@ -0,0 +1,210 @@ +# Design brief: Modelith animated demo (code-based, for `/design`) + +Paste this whole brief into `/design`. It builds **one canvas with 3 artboards**, each an +**animated terminal + ER diagram** demo of one Modelith story. Everything below is real — +real `mdl` commands, real output, real entities — captured from the working CLI. Do not invent +commands or output; use exactly what's here. + +## Ground it in the REAL repo (read these first) + +This animation must reproduce the **real Modelith Studio canvas**, not an approximation. Before +building, read these in the repo: + +- **`demos/design-assets/canvas-reference.jpg`** — a real screenshot of the live Studio canvas + (the IBOR model). This is the visual target: the exact entity-card look, the amber top border, + the gold-key PK rows, the dim mono types, the dashed crow's-foot relationship lines, the + warm-charcoal dotted background. Match it. +- **`demos/design-assets/canvas-tokens.css`** — the REAL design tokens and entity-card CSS, + extracted verbatim from `canvas/src/styles.css`. Use these exact values and class structure + for the ER cards in the animation. +- For deeper fidelity, the real source is `canvas/src/styles.css` (the `:root` tokens, + `.entity-node`, `.entity-header`, `.badge`, `.attr-row`) and `canvas/src/EntityNode.tsx` + (the card markup: header + badges, a keys section, then the attribute rows). + +The terminal panel is a plain dark terminal (same background/mono font); it doesn't need to copy +a specific app chrome. + +--- + +## What to build + +A horizontal filmstrip canvas, **3 artboards** (16:9, 1920×1080 each), named: +`Main.dc.html` = **1-Design-first**, `Adopt.dc.html` = **2-Adopt-a-warehouse**, +`Drift.dc.html` = **3-Drift-gate**. + +Each artboard is a **split scene**: a terminal panel on the LEFT (~55% width) and a live +**ER diagram** on the RIGHT (~45%). The terminal types commands and reveals output; the ER +diagram animates in sync (entities appear, columns highlight, a change folds in). Animations +loop or play once on view — your call; prefer a ~12–18s timed sequence per artboard that loops. + +## Visual identity (match the Modelith product) + +- Background `#14110f` (warm charcoal). Panels `#1c1815`, cards `#221d19`, borders `#33291f`. +- Text `#ece6dd`, dim text `#a89a86`. **Accent amber `#e0a63a`** (the brand color). +- Success green `#6cbf73`, warn/amber `#e6a94a`, error `#f0857a`, PK gold `#f2c14e`. +- Monospace for the terminal: "JetBrains Mono", ui-monospace, Menlo. Sans for labels: Inter. +- Terminal prompt is `$`. Typed commands animate character-by-character (~40ms/char feel). + Output reveals as a block after a short beat. Keep a blinking amber cursor. +- ER entity cards: rounded, dark card, a 3px amber top border, a bold entity name header, then + attribute rows (name left, type right). PK rows show a gold `PK` chip; FK rows an amber `FK` + chip. Relationship lines connect FK column to the referenced entity (crow's-foot if easy). + +--- + +## Artboard 1 — "Design-first" (greenfield) + +**Tagline (small, top):** "Design in git. Compile to contract-enforced dbt." + +**Terminal sequence (type + reveal, in order):** + +``` +$ mdl model render +# model: shop +entity customer customer_id [pk], email, full_name, created_at +entity order order_id [pk], customer_id, order_date, status +entity order_line order_line_id [pk], order_id, product_id, quantity +entity product product_id [pk], sku, name, unit_price + +$ mdl new entity payment --definition "A settlement against an order" +created entity 'payment' (01M3WX64NZ...) + +$ mdl validate +validation passed + +$ mdl generate +wrote 9 files to target/dbt + +$ mdl export contract +wrote contract to datacontract.yaml +``` + +**ER diagram animation (synced):** +- As `model render` reveals, the 4 entity cards **fade/slide in**: `customer`, `order`, + `order_line`, `product`. Draw FK relationship lines: `order.customer_id → customer`, + `order_line.order_id → order`, `order_line.product_id → product`. +- On `mdl new entity payment`, a **5th card `payment` animates in** (amber glow pulse), with a + `payment_id [pk]` row (it's a fresh entity, minimal attrs). +- On `mdl validate` → a green "✓ valid" badge pulses over the diagram. +- On `mdl generate` / `export contract` → two small output chips slide in at the bottom: + a dbt logo-ish chip "dbt • contract enforced" and a "ODCS data contract" chip — the + "one model, many targets" idea. Keep subtle. + +**Entity/attribute data (use verbatim):** +- customer: customer_id (bigint, PK), email (string), full_name (string), created_at (timestamp) +- order: order_id (bigint, PK), customer_id (bigint, FK→customer), order_date (date), status (string) +- order_line: order_line_id (bigint, PK), order_id (bigint, FK→order), product_id (bigint, FK→product), quantity (integer) +- product: product_id (bigint, PK), sku (string), name (string), unit_price (decimal) +- payment: payment_id (bigint, PK) [appears on `new entity`] + +--- + +## Artboard 2 — "Adopt a warehouse" (brownfield / reverse) + +**Tagline:** "Point it at a legacy warehouse. Get a clean model back." + +**Terminal sequence:** + +``` +$ ls models/staging models/marts models/vault +stg_customers.sql stg_orders.sql int_order_enriched.sql +dim_customer.sql dim_product.sql fct_orders.sql +hub_customer.sql link_order.sql sat_customer_details.sql + +$ mdl reverse --project target/manifest.json --out owned +reversed 6 entities (4 staging/intermediate excluded) + kept as business entities (4): dim_customer, dim_product, fct_orders, hub_customer + excluded as staging/intermediate (4): stg_customers, stg_orders, ... + SCD2 pattern detected (2): dim_customer, dim_product + Data Vault detected (3): hub_customer, link_order, sat_customer_details + surrogate keys stripped (7) + +$ mdl validate -m owned +validation passed +``` + +**Reverse output must keep the color coding:** "kept as business entities" in GREEN, "managed +but no business key" in AMBER, "excluded" in BLUE-ish, "SCD2 / Data Vault" headers in blue, +"surrogate keys stripped" header in blue. This color-coded classification is the hero of the shot. + +**ER diagram animation:** +- Start with a **messy pile** of raw SQL file chips (stg_*, dim_*, fct_*, hub_*, link_*, sat_*) + scattered/greyed on the right. +- On `mdl reverse`, animate the transform: staging chips **fade out** (excluded); the business + entities **snap into clean ER cards** — `dim_customer [SCD2]`, `dim_product [SCD2]`, + `fct_orders`, `hub_customer [HUB]` — with `[SCD2]`/`[HUB]`/`[LINK]` badge tags. Draw FK lines: + `fct_orders.customer_id → dim_customer`, `fct_orders.product_id → dim_product`. +- Small floating annotations pulse as each pattern is "detected": a tag "SCD2" on dim_customer, + "Data Vault" on the hub/link/sat group, "surrogate key stripped" briefly on a removed `_sk` + column. These mirror the real detection. + +**Entity/attribute data (verbatim from the real reverse):** +- dim_customer [SCD2]: customer_id (integer, PK), name (string), tier (string) +- dim_product [SCD2]: product_id (integer, PK), sku (string), category (string), list_price (decimal) +- fct_orders: order_id (integer, PK), customer_id (FK→dim_customer), product_id (FK→dim_product), order_date (date), amount (decimal) +- hub_customer [HUB]: customer_id (integer, PK), load_dts (date) +- link_order [LINK], sat_customer_details — shown as tagged cards, fewer attrs + +--- + +## Artboard 3 — "The drift gate" (the climax) + +**Tagline:** "The model and the warehouse, always in lockstep." + +**Terminal sequence (this is the emotional peak — pace it):** + +``` +$ mdl drift -m model --manifest target/manifest.json +no drift detected + +# a teammate adds a broker_code column to the warehouse +$ mdl drift -m model --manifest target/manifest.json --check +drift vs target 'duckdb_dev': + [additive] 1 + - column transaction.broker_code exists in dbt but not the model + +$ mdl drift -m model --manifest target/manifest.json --reconcile +reconciled 1 delta(s); skipped 0 breaking + + add column transaction.broker_code + +$ mdl drift -m model --manifest target/manifest.json +no drift detected +``` + +**Color:** `no drift detected` neutral/green; `[additive] 1` in AMBER; `reconciled 1 delta(s); +skipped 0 breaking` in GREEN; `+ add column transaction.broker_code` with a green `+`. + +**ER diagram animation (the key moment):** +- Center the **`transaction`** entity card: transaction_id [PK], trade_date, amount, + portfolio_code [FK→portfolio], counterparty_id [FK→counterparty]. (Optionally small + portfolio + counterparty cards to its sides with FK lines.) +- Scene 1 — a steady state; a subtle "model ⇄ warehouse in sync" indicator (two panels linked, + green). +- Scene 2 — a **new `broker_code` row** appears on the WAREHOUSE side only (right), glowing + AMBER, with a floating "additive" tag. The sync indicator turns amber. This is drift. +- Scene 3 — on `--reconcile`, the `broker_code` row **animates from the warehouse side INTO the + model's `transaction` card** (slides over, snaps in, turns from amber to normal). The sync + indicator returns to GREEN. A small "nothing lost" / "✓ reconciled" note pulses. +- End on both sides matching, green, calm. + +**Entity data (verbatim):** +- transaction: transaction_id (string, PK), trade_date (date), amount (decimal), + portfolio_code (string, FK→portfolio), counterparty_id (bigint, FK→counterparty) +- the added column: broker_code (string) — additive, folded in by reconcile. + +--- + +## Overall feel + +- Calm, confident, "engineering-grade" — not flashy. Amber accent used sparingly for emphasis + (the cursor, a badge, a glow on the key moment). Everything else restrained and dark. +- Each artboard should read in one glance (what story) and reward watching (the animation). +- No fake UI chrome that impersonates another product. The terminal + ER diagram are generic + Modelith surfaces. +- If sound/voice is off the table (it is, in /design), let the on-screen taglines + the typed + commands carry the narrative. + +## Source of truth + +These flows are verified real. The matching VHS reference clips and the exact `.tape` scripts +are under `demos/tapes/films/`, and the live commands run against `demos/fixtures/shop`, +`demo/legacy-warehouse`, and `demo/ibor`. Nothing here is mocked. diff --git a/demos/README.md b/demos/README.md new file mode 100644 index 0000000..b86212f --- /dev/null +++ b/demos/README.md @@ -0,0 +1,72 @@ +# Modelith CLI demos + +Three short **films** of the `mdl` CLI, each a realistic project followed end to end as one +continuous session — the project is set up once and evolves on screen, no mid-story resets. +Scripted with [VHS](https://github.com/charmbracelet/vhs) (not screen-captured) so every take +is identical, typo-free, re-renderable when the CLI changes, and visually consistent. + +**Everything is real** — real `mdl` commands, real `dbt parse`, real output. Changes are shown +on-camera; nothing is faked or staged. + +## The three films + +### 1 — Design-first (greenfield, ~45s) +From a blank repo to running dbt + a data contract. +`mdl model render` (the designed model) → `mdl new entity` (mints ULIDs) → `mdl validate` → +`mdl generate` (protected regions + an ENFORCED contract) → `mdl export contract` (one model → +an ODCS data contract too). **Story: design in git, compile to more than dbt.** + +### 2 — Adopt a warehouse (brownfield, ~55s) +Take over a legacy dbt project that has no data model. +`ls` the warehouse (staging / marts / vault) → `mdl reverse` (real SCD2 / Data Vault / +surrogate-key detection) → `mdl decisions list` (every inference, confidence-tagged, reviewable) +→ `mdl validate` → `mdl model render` (clean owned model). **Story: inherit a mess, own it in +one command, nothing is a black box.** + +### 3 — The drift gate (the climax, ~60s) +The model and the warehouse stay in lockstep. +`mdl drift` (clean) → a teammate adds a `broker_code` column to a dbt model + its contract (a +real edit) → `dbt parse` → `mdl drift --check` (real: classified **additive**) → `mdl drift +--reconcile` (real: folds the safe change into the model) → `mdl drift` (clean) + `mdl validate`. +**Story: drift is caught and classified; safe change folds in; work is never lost.** + +The Studio **canvas** is a browser UI, not terminal output — recorded separately as a browser +screen capture, not a VHS tape. (Spotlight clips for structural merge, multi-target compile, and +the AI/MCP surface can be added the same way these films are built.) + +## Rendering + +Requirements (macOS): + +```bash +brew install vhs ffmpeg +brew install --cask font-jetbrains-mono +uv sync # so .venv/bin/mdl exists — the tapes put it on PATH for a fast, clean `mdl` +``` + +Then: + +```bash +demos/render.sh # render all films → demos/out/*.mp4 +demos/render.sh 3 # just film 3 +``` + +## How the tapes work + +- **`tapes/_settings.tape`** — shared look: JetBrains Mono 22px, 1280×900 (tall enough that a + continuous session doesn't scroll earlier scenes off the top), the Modelith amber theme, 70ms + typing. `Source`d by every film; tune once here. +- Each film has ONE **hidden setup block** (`Hide … Show`) that puts a fast `mdl` on PATH from + the project venv and copies the film's fixture to a throwaway `/tmp/mdl-film*` dir. After that + the whole arc runs in one session; the on-screen `clear`s are scene breaks, not resets. +- `setopt interactivecomments` lets the narration `# …` lines show as comments in zsh. +- Timing is **fixed `Sleep`s** sized to the CLI's ~2s startup plus a reading beat (not `Wait`, + which raced the slow start and echoed keystrokes). +- Fixtures: the bundled `demo/` models (ibor, legacy-warehouse), the greenfield `fixtures/shop` + model, and `fixtures/add-broker-code.sh` (the real warehouse edit for film 3). + +## Re-rendering when the CLI changes + +The films drive the real `mdl` + `dbt`, so a change can alter the output. After upgrading: +`uv sync`, then `demos/render.sh`, and watch the new films. If a command's flags changed, edit +the matching `.tape` (commands are plain text) and re-render just that one. diff --git a/demos/design-assets/canvas-reference.jpg b/demos/design-assets/canvas-reference.jpg new file mode 100644 index 0000000..bafda7c Binary files /dev/null and b/demos/design-assets/canvas-reference.jpg differ diff --git a/demos/design-assets/canvas-tokens.css b/demos/design-assets/canvas-tokens.css new file mode 100644 index 0000000..1e4e963 --- /dev/null +++ b/demos/design-assets/canvas-tokens.css @@ -0,0 +1,63 @@ +/* REAL design tokens + entity-card CSS, extracted verbatim from the Modelith Studio canvas + (canvas/src/styles.css). Use these EXACT values in the animation so the ER diagram matches + the real product. The screenshot canvas-reference.jpg shows how these render live. */ + +:root { + --bg: #14110f; /* warm charcoal background */ + --bg-raised: #1c1815; /* panels, toolbar */ + --bg-card: #221d19; /* entity cards, inputs */ + --border: #33291f; + --border-strong: #4a3c2c; + --text: #ece6dd; + --text-dim: #a89a86; + --accent: #e0a63a; /* brand amber */ + --accent-soft: rgba(224, 166, 58, 0.15); + --accent-ink: #241a08; + --pk: #f2c14e; /* primary-key gold */ + --err: #f0857a; + --warn: #e6a94a; + --ok: #6cbf73; /* success green */ + --font: "Inter", -apple-system, "Segoe UI", Roboto, sans-serif; + --mono: "JetBrains Mono", ui-monospace, "Cascadia Code", Menlo, monospace; +} + +/* --- the real entity card (ER node) --- */ +.entity-node { + width: 264px; background: var(--bg-card); + border: 1px solid var(--border); + border-top: 3px solid var(--accent); /* amber top edge — the signature */ + border-radius: 10px; overflow: hidden; + box-shadow: 0 4px 18px rgba(0, 0, 0, 0.35); + font-size: 12px; +} +.entity-header { + display: flex; align-items: center; justify-content: space-between; + padding: 10px 12px; background: rgba(255, 255, 255, 0.03); + border-bottom: 1px solid var(--border); +} +.entity-name { font-weight: 700; font-size: 13px; letter-spacing: 0.01em; color: var(--text); } +.entity-badges { display: flex; gap: 4px; } +/* pattern badge: SCD2 / HUB / LINK / SAT tags (amber pill). ontology badge is blue. */ +.badge { + font-size: 10px; padding: 1px 6px; border-radius: 99px; + background: var(--accent-soft); color: var(--accent); + text-transform: uppercase; letter-spacing: 0.05em; font-weight: 600; +} +.badge.ontology { background: rgba(147, 197, 253, 0.12); color: #93c5fd; } + +/* the PK/FK key rows sit in their own faintly-gold section at the top of the card body */ +.attr-section.keys { border-bottom: 1px solid var(--border); background: rgba(252, 211, 77, 0.04); } +.attr-section { padding: 4px 0; } +.attr-row { + display: flex; align-items: center; gap: 6px; + padding: 3px 12px; height: 26px; line-height: 20px; +} +.attr-icon { width: 14px; text-align: center; font-size: 10px; flex: none; } /* role glyph, e.g. a key for PK */ +.attr-name { flex: 1; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; color: var(--text); } +.attr-name.pk { color: var(--pk); font-weight: 600; } /* primary key rows render in gold */ +.attr-type { color: var(--text-dim); font-family: var(--mono); font-size: 10px; flex: none; } /* type, right-aligned, mono, dim */ + +/* relationship lines between cards: thin, dim, dashed; crow's-foot at the "many" end */ +.rel-path { stroke: var(--border-strong); stroke-width: 1.6; } /* in the live canvas these are dashed amber-grey */ + +/* the canvas background is var(--bg) with a faint dot grid (#33291f dots, ~22px gap) */ diff --git a/demos/fixtures/add-broker-code.sh b/demos/fixtures/add-broker-code.sh new file mode 100644 index 0000000..83a4363 --- /dev/null +++ b/demos/fixtures/add-broker-code.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# A teammate adds a broker_code column to the transaction model and its contract. +set -e +sql=transform/warehouse/models/transaction.sql +schema=transform/warehouse/models/schema.yml + +# add the column to the model's SELECT (a real dbt column) +perl -0pi -e 's/( counterparty_id\n)(from source\n-- mdl:generated-end)/$1 , cast(null as varchar) as broker_code\n$2/' "$sql" + +# declare it in the enforced contract (so dbt knows its type) +python3 - "$schema" <<'PY' +import sys +f = sys.argv[1] +s = open(f).read() +i = s.find("- name: transaction\n") +j = s.find("columns:\n", i) + len("columns:\n") +block = (" - name: broker_code\n" + " data_type: VARCHAR\n" + " meta:\n" + " mdl_ulid: 01M3XBROKERCODE0000000000X\n") +open(f, "w").write(s[:j] + block + s[j:]) +PY +echo "warehouse: + transaction.broker_code (SQL + contract)" diff --git a/demos/fixtures/shop/.gitignore b/demos/fixtures/shop/.gitignore new file mode 100644 index 0000000..ef142c7 --- /dev/null +++ b/demos/fixtures/shop/.gitignore @@ -0,0 +1,9 @@ +# dbt build artifacts (generated projects live here by default) +target/ +dbt_packages/ +logs/ +# Fetched ontology layers — pinned by .mdl/lock.yaml, not committed (spec §3). +# Run `mdl ontology fetch` to repopulate. Like node_modules to package-lock.json. +.mdl/ontology-cache/ +# Reverse/generation working state (the committed lock/decisions stay tracked). +.mdl/state/ diff --git a/demos/fixtures/shop/.mdl/lock.yaml b/demos/fixtures/shop/.mdl/lock.yaml new file mode 100644 index 0000000..84c4e63 --- /dev/null +++ b/demos/fixtures/shop/.mdl/lock.yaml @@ -0,0 +1,6 @@ +dbt: '1.9' +osi: + tag: osi-0.1.1-rc1 + version: 0.1.1 +vocabularies: {} +profiles: {} diff --git a/demos/fixtures/shop/conceptual/entities/customer.yaml b/demos/fixtures/shop/conceptual/entities/customer.yaml new file mode 100644 index 0000000..416d784 --- /dev/null +++ b/demos/fixtures/shop/conceptual/entities/customer.yaml @@ -0,0 +1,3 @@ +id: 01M3WS6GDGNDJSQ8J69WPTEC02 +kind: conceptual_entity +name: Customer diff --git a/demos/fixtures/shop/conceptual/entities/order.yaml b/demos/fixtures/shop/conceptual/entities/order.yaml new file mode 100644 index 0000000..7b30bf8 --- /dev/null +++ b/demos/fixtures/shop/conceptual/entities/order.yaml @@ -0,0 +1,3 @@ +id: 01M3WS6MYM4HWTS7Z022BFS98T +kind: conceptual_entity +name: Order diff --git a/demos/fixtures/shop/conceptual/entities/order_line.yaml b/demos/fixtures/shop/conceptual/entities/order_line.yaml new file mode 100644 index 0000000..651d06c --- /dev/null +++ b/demos/fixtures/shop/conceptual/entities/order_line.yaml @@ -0,0 +1,3 @@ +id: 01M3WS6Q5Y8C3J4GH1B3Q3TCDR +kind: conceptual_entity +name: OrderLine diff --git a/demos/fixtures/shop/conceptual/entities/product.yaml b/demos/fixtures/shop/conceptual/entities/product.yaml new file mode 100644 index 0000000..efff24c --- /dev/null +++ b/demos/fixtures/shop/conceptual/entities/product.yaml @@ -0,0 +1,3 @@ +id: 01M3WS6JBQ4NRR17WNYA2N7527 +kind: conceptual_entity +name: Product diff --git a/demos/fixtures/shop/logical/domains/identifier_bigint.yaml b/demos/fixtures/shop/logical/domains/identifier_bigint.yaml new file mode 100644 index 0000000..b89b16f --- /dev/null +++ b/demos/fixtures/shop/logical/domains/identifier_bigint.yaml @@ -0,0 +1,5 @@ +id: 01M3WS6D96RM44FGPXP6CQ52VA +kind: domain +name: identifier_bigint +base_type: bigint +definition: A surrogate or business identifier stored as a 64-bit integer. diff --git a/demos/fixtures/shop/logical/entities/customer.yaml b/demos/fixtures/shop/logical/entities/customer.yaml new file mode 100644 index 0000000..1c642eb --- /dev/null +++ b/demos/fixtures/shop/logical/entities/customer.yaml @@ -0,0 +1,25 @@ +id: 01M3WS6GDGEEH1Y8WHQ6MV3V3F +kind: logical_entity +name: customer +realises: 01M3WS6GDGNDJSQ8J69WPTEC02 +attributes: +- id: 01NHT44FHT243W0EC7NYE7XT5G + name: customer_id + domain: identifier_bigint + role: business_key + nullable: false +- id: 019KNST7XJ2DKXTC87G0EA3HKH + name: email + domain: string + role: attribute + nullable: false +- id: 019GDP66FGFQKT68CFVR3ZM8FQ + name: full_name + domain: string + role: attribute + nullable: false +- id: 01QP2BSPBXG463YDY605ZJ69J9 + name: created_at + domain: timestamp + role: attribute + nullable: false diff --git a/demos/fixtures/shop/logical/entities/order.yaml b/demos/fixtures/shop/logical/entities/order.yaml new file mode 100644 index 0000000..d20dd5e --- /dev/null +++ b/demos/fixtures/shop/logical/entities/order.yaml @@ -0,0 +1,25 @@ +id: 01M3WS6MYM4WC27B98H8N0FMEE +kind: logical_entity +name: order +realises: 01M3WS6MYM4HWTS7Z022BFS98T +attributes: +- id: 01FRDRCCE2C2AE7XW43T7G043M + name: order_id + domain: identifier_bigint + role: business_key + nullable: false +- id: 014BKSP4BRDZZSKSTC1T5T3SVK + name: customer_id + domain: identifier_bigint + role: attribute + nullable: false +- id: 01A7ZKYFT9P1Y63R1W01RC442D + name: order_date + domain: date + role: attribute + nullable: false +- id: 010BHNZSXXY1G3J01ZSR27NP9R + name: status + domain: string + role: attribute + nullable: false diff --git a/demos/fixtures/shop/logical/entities/order_line.yaml b/demos/fixtures/shop/logical/entities/order_line.yaml new file mode 100644 index 0000000..bdfdc6b --- /dev/null +++ b/demos/fixtures/shop/logical/entities/order_line.yaml @@ -0,0 +1,25 @@ +id: 01M3WS6Q5YM32P4ZWX3DQQF4TG +kind: logical_entity +name: order_line +realises: 01M3WS6Q5Y8C3J4GH1B3Q3TCDR +attributes: +- id: 01E27JA246C69XXSNHG3G5NXSM + name: order_line_id + domain: identifier_bigint + role: business_key + nullable: false +- id: 01WCBNMDRE9XY1MFNG29G1XKHH + name: order_id + domain: identifier_bigint + role: attribute + nullable: false +- id: 01EBXNY3SSNY6DVMFR1W3KQQVS + name: product_id + domain: identifier_bigint + role: attribute + nullable: false +- id: 01M260C4FKNHRBXRFKY69QG3R3 + name: quantity + domain: integer + role: attribute + nullable: false diff --git a/demos/fixtures/shop/logical/entities/product.yaml b/demos/fixtures/shop/logical/entities/product.yaml new file mode 100644 index 0000000..5057fa5 --- /dev/null +++ b/demos/fixtures/shop/logical/entities/product.yaml @@ -0,0 +1,25 @@ +id: 01M3WS6JBQXS2JC5461EEYEZJD +kind: logical_entity +name: product +realises: 01M3WS6JBQ4NRR17WNYA2N7527 +attributes: +- id: 01A3Y3VYBXYFQKQP4CBWFG7NY9 + name: product_id + domain: identifier_bigint + role: business_key + nullable: false +- id: 0146BZKSG0FJ1T646003JCC9NK + name: sku + domain: string + role: attribute + nullable: false +- id: 01KG5MDJ2A5Y3RDSHQZNGFRBXT + name: name + domain: string + role: attribute + nullable: false +- id: 0121ZQHKJC7J9SQKXPA9SQY3ME + name: unit_price + domain: decimal + role: attribute + nullable: false diff --git a/demos/fixtures/shop/mdl-project.yaml b/demos/fixtures/shop/mdl-project.yaml new file mode 100644 index 0000000..c4e3277 --- /dev/null +++ b/demos/fixtures/shop/mdl-project.yaml @@ -0,0 +1,37 @@ +# Modelith project config (spec §2.2) +name: shop +dbt_target: duckdb_dev +platform_targets: + - duckdb_dev +# Ontology sources plug in by declaration. A source is either a local file +# vocabulary (default type: local) or a remote resolver browsed live for +# autocomplete/search (types: ols, ols-compatible, ontoportal, collibra). FIBO is +# only one example — ACORD / FHIR / ISO 20022 / your own plug in the same way. +# ontology_stack: +# - name: fibo # local file vocabulary +# layer: industry +# format: turtle +# path: ontologies/industry/fibo/2024.03 +# modules: [fnd, fbc] +# prefixes: +# fibo-fnd-pty-pty: "https://spec.edmcouncil.org/fibo/ontology/FND/Parties/Parties/" +# - name: ols # public OLS4 (no auth) — live search only +# layer: industry +# type: ols +# url: https://www.ebi.ac.uk/ols4/api +# - name: bioportal # OntoPortal/BioPortal (needs an API key) +# layer: domain +# type: ontoportal +# url: https://data.bioontology.org +# apikey_env: BIOPORTAL_APIKEY +# - name: collibra # Collibra Ontology Domains (bearer token) +# layer: core +# type: collibra +# url: https://acme.collibra.com +# token_env: COLLIBRA_TOKEN +# domain_types: [Ontology] +ontology_stack: [] +naming: + logical_case: snake + physical_case: upper_snake + abbreviations: {} diff --git a/demos/landing/index.html b/demos/landing/index.html new file mode 100644 index 0000000..5b6312c --- /dev/null +++ b/demos/landing/index.html @@ -0,0 +1,261 @@ + + + + + +Modelith — the modeling layer dbt-core is missing + + + + + + + + + + + + +
+ git-native · ontology-anchored · dbt-core +

The modeling layer dbt-core is missing

+

Design entities and relationships in git, generate contract-enforced dbt, and + catch drift before it ships. One model every tool reads and writes — the CLI, the canvas, + VS Code, and CI.

+ +
+
# no cloud warehouse needed — the demo builds on a bundled DuckDB
+
$ uv tool install 'modelith-dbt[ontology]'
+
$ mdl init my-model
+
$ mdl generate # → contract-enforced dbt
+
+
+ +
+
See it
+

Six flows, end to end

+

Every command and every line of output below is real. Nothing is mocked.

+ +
+ +
+
+
+
01 · Design-first
+

From a blank repo to running dbt

+

Scaffold a model, add entities (ULIDs minted for you), validate, and compile to + contract-enforced dbt — plus an ODCS data contract from the same model.

+ mdl init · new entity · validate · generate +
+
+ +
+
+
+
02 · Adopt a warehouse
+

Reverse a legacy dbt project into a clean model

+

Point it at a warehouse with no data model. It excludes staging, detects SCD2 and + Data Vault, strips surrogate keys, and logs every inference in a reviewable ledger.

+ mdl reverse --project target/manifest.json +
+
+ +
+
+
+
03 · The drift gate
+

Model and warehouse, always in lockstep

+

A column lands in the warehouse. Drift classifies it — additive, not breaking — + and reconcile folds the safe change into the model. Nothing is ever lost.

+ mdl drift --check · --reconcile +
+
+ +
+
+
+
04 · Ontology anchoring
+

Align to an industry vocabulary

+

Search FIBO (or any OLS / OntoPortal / Collibra source), align an entity to a + standard term, and track coverage — anchored, reproducible, offline.

+ mdl ontology search · align · status +
+
+ +
+
+
+
05 · Erwin import
+

Bring an erwin model into git

+

Import an erwin XML export straight into clean logical YAML you can version, diff, + and generate from. The migration path off a legacy modeling tool.

+ mdl import erwin fund.xml -o fund +
+
+ +
+
+
+
06 · One model, many targets
+

A contract factory

+

From one definition: contract-enforced dbt, an ODCS data contract, Pydantic models, + SQL DDL, Mermaid, DBML, RDF/SHACL, and a Neo4j graph schema — each deterministic.

+ mdl generate · export · emit +
+
+ +
+
+ +
+
Why
+

Meaning lives in one place, in git

+

Warehouse teams scatter their meaning across an ER tool that never sees + production, dbt SQL that drifts from the design, and a catalog nobody edits. Modelith + collapses them into one model every surface reads and writes.

+
+

Contract-enforced dbt

Generate dbt-core models with enforced + contracts, key constraints, and relationship tests. Regeneration runs a three-way merge, + so hand edits and generated blocks both survive.

+

Drift as a CI gate

Compare the committed model to the compiled + warehouse. Breaking / additive / cosmetic, classified in seconds. --check + fails the PR; --reconcile folds the safe deltas back in.

+

Reverse any warehouse

A live database, a dbt manifest, or raw + SQL DDL, lifted into a clean model — SCD2 and Data Vault detected, surrogates stripped, + every inference reviewable.

+

Ontology-anchored

A four-layer vocabulary stack with SKOS + alignments and provenance. Pin bundles like dependencies; resolve against OLS, OntoPortal, + or Collibra — reproducibly, offline.

+

Structural git merge

A ULID-keyed merge driver: two people add + different attributes to the same entity and merge clean. A rename is one cosmetic change, + not a delete-plus-add.

+

Every surface is a client

The model is plain YAML in git. The + CLI, the web canvas, the VS Code extension, and CI all read and write it — none of them + owns state.

+
+
+ +
+
Model once, ship many
+

One model compiles to every target it needs

+

Each generated deterministically from the single definition.

+
+
dbt-coremdl generate
+
ODCS data contractmdl export contract
+
Pydanticmdl emit pydantic
+
Neo4j graphmdl export graph
+
Semantic layermdl emit semantic
+
RDF / SHACLmdl export rdf
+
SQL DDLmdl export sql
+
Mermaid ERDmdl export mermaid
+
DBMLmdl export dbml
+
+
+ +
+

Try it in two minutes

+

No cloud warehouse. The IBoR demo ships a model and + a dbt project over a bundled DuckDB, so it builds on a laptop.

+ +
+ + + + + diff --git a/demos/landing/media/Adopt.mp4 b/demos/landing/media/Adopt.mp4 new file mode 100644 index 0000000..81a9d2e Binary files /dev/null and b/demos/landing/media/Adopt.mp4 differ diff --git a/demos/landing/media/Compile.mp4 b/demos/landing/media/Compile.mp4 new file mode 100644 index 0000000..5d97080 Binary files /dev/null and b/demos/landing/media/Compile.mp4 differ diff --git a/demos/landing/media/Drift.mp4 b/demos/landing/media/Drift.mp4 new file mode 100644 index 0000000..49916c6 Binary files /dev/null and b/demos/landing/media/Drift.mp4 differ diff --git a/demos/landing/media/Erwin.mp4 b/demos/landing/media/Erwin.mp4 new file mode 100644 index 0000000..21d47cd Binary files /dev/null and b/demos/landing/media/Erwin.mp4 differ diff --git a/demos/landing/media/Main.mp4 b/demos/landing/media/Main.mp4 new file mode 100644 index 0000000..fb0524e Binary files /dev/null and b/demos/landing/media/Main.mp4 differ diff --git a/demos/landing/media/Ontology.mp4 b/demos/landing/media/Ontology.mp4 new file mode 100644 index 0000000..297a6b4 Binary files /dev/null and b/demos/landing/media/Ontology.mp4 differ diff --git a/demos/render.sh b/demos/render.sh new file mode 100755 index 0000000..ba7770d --- /dev/null +++ b/demos/render.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Render the Modelith CLI demo tapes to MP4 under demos/out/. +# +# Requirements: vhs, ffmpeg, the JetBrains Mono font, and the project venv: +# brew install vhs ffmpeg && brew install --cask font-jetbrains-mono && uv sync +# The tapes put a fast `mdl` on PATH from .venv/bin so recorded commands start quickly and +# show a clean `mdl`. +# +# Usage: +# demos/render.sh # every tape +# demos/render.sh a-ibor # one series (matches the path) +# demos/render.sh a3 # one chapter (matches the filename) +set -euo pipefail + +cd "$(dirname "$0")/.." # repo root, so each tape's demos/... paths resolve + +filter="${1:-}" +shopt -s nullglob +tapes=(demos/tapes/films/[0-9]*.tape) + +for tape in "${tapes[@]}"; do + if [[ -n "$filter" && "$tape" != *"$filter"* ]]; then continue; fi + echo "▶ rendering $tape" + vhs "$tape" +done + +echo "✓ done — outputs in demos/out/" +ls -lh demos/out/*.mp4 2>/dev/null | awk '{print $9, $5}' || true diff --git a/demos/standalone/Adopt.html b/demos/standalone/Adopt.html new file mode 100644 index 0000000..322195a --- /dev/null +++ b/demos/standalone/Adopt.html @@ -0,0 +1,197 @@ + + + + + +Adopt a warehouse + + + + +
+ +
+ ◮ + Modelith + reverse + Point it at a legacy warehouse. Get a clean model back. +
+ +
+ + +
+
+ + + + mdl — legacy-warehouse +
+
+
$ ls models/staging models/marts models/vault
+
stg_customers.sql dim_customer.sql hub_customer.sql
+
stg_orders.sql fct_orders.sql link_order.sql sat_customer_details.sql
+
+
$ mdl reverse --project target/manifest.json --out owned
+
reversed 6 entities (4 staging/intermediate excluded)
+
kept as business entities (4)
+
dim_customer, dim_product, fct_orders, hub_customer
+
excluded as staging/intermediate (4)
+
stg_customers, stg_orders, int_order_enriched, …
+
SCD2 pattern detected (2)
+
dim_customer, dim_product
+
Data Vault detected (3)
+
hub_customer, link_order, sat_customer_details
+
surrogate keys stripped (7)
+
+
$ mdl validate -m owned
+
validation passed
+
$
+
+
+ + +
+
+ + +
stg_customers.sql
+
stg_orders.sql
+
int_order_enriched.sql
+
dim_customer.sql
+
fct_orders.sql
+
hub_customer.sql
+
link_order.sql
+
sat_customer_details.sql
+ + + + + + + +
+
dim_customerSCD2
+
+
🔑 customer_idinteger
+
namestring
+
tierstring
+
+
+ +
+
dim_productSCD2
+
+
🔑 product_idinteger
+
skustring
+
categorystring
+
list_pricedecimal
+
+
+ +
+
fct_orders
+
+
🔑 order_idinteger
+
customer_id FKinteger
+
product_id FKinteger
+
amountdecimal
+
+
+ +
+
hub_customerHUB
+
+
🔑 customer_idinteger
+
load_dtsdate
+
+
+ +
+
link_orderLINK
+
+
order_hashkeystring
+
load_dtsdate
+
+
+
+
+
+
+ + + \ No newline at end of file diff --git a/demos/standalone/Compile.html b/demos/standalone/Compile.html new file mode 100644 index 0000000..aae6aae --- /dev/null +++ b/demos/standalone/Compile.html @@ -0,0 +1,171 @@ + + + + + +One model, many targets + + + + +
+ +
+ ◮ + Modelith + compile + One model. Compiled to every target it needs. +
+ +
+ + +
+
+ + + + mdl — ibor +
+
+
$ mdl generate
+
wrote 11 files to target/dbt
+
$ mdl export contract
+
wrote contract to datacontract.yaml
+
$ mdl emit pydantic
+
wrote pydantic to models.py
+
$ mdl export sql --dialect postgres
+
-- SQL DDL (postgres)
+
$ mdl export mermaid
+
erDiagram …
+
$ mdl export dbml
+
// DBML …
+
$
+
+
+ + +
+
+ + + + + + + + + + + + +
+
modelone source of truth
+
+
entities7
+
relationships8
+
contractsenforced
+
+
+ + +
+
dbt-core
+
contract-enforced models
+
+
+
ODCS contract
+
datacontract.yaml
+
+
+
Pydantic
+
models.py
+
+
+
SQL DDL
+
CREATE TABLE · any dialect
+
+
+
Mermaid ERD
+
renders in GitHub
+
+
+
DBML
+
dbdiagram.io
+
+
+
+
+
+ + + \ No newline at end of file diff --git a/demos/standalone/Drift.html b/demos/standalone/Drift.html new file mode 100644 index 0000000..2777b36 --- /dev/null +++ b/demos/standalone/Drift.html @@ -0,0 +1,199 @@ + + + + + +The drift gate + + + + +
+ +
+ ◮ + Modelith + drift + The model and the warehouse, always in lockstep. +
+ +
+ + +
+
+ + + + mdl — ibor +
+
+
$ mdl drift -m model --manifest …/manifest.json
+
no drift detected
+
+
# a teammate adds a broker_code column to the warehouse
+
$ mdl drift … --check
+
drift vs target 'duckdb_dev':
+
[additive] 1
+
- column transaction.broker_code exists in dbt but not the model
+
+
$ mdl drift … --reconcile
+
reconciled 1 delta(s); skipped 0 breaking
+
+ add column transaction.broker_code
+
+
$ mdl drift -m model --manifest …/manifest.json
+
no drift detected
+
$
+
+
+ + +
+ + +
+ +
+ +
+ + +
+
model
+
+
transaction
+
+
🔑 transaction_idstring
+
trade_datedate
+
amountdecimal
+
portfolio_code FKstring
+
counterparty_id FKbigint
+ +
+
broker_codestring
+
+
+
+
+ + + + + + + +
+
warehouse (dbt)
+
+
transaction
+
+
🔑 transaction_idstring
+
trade_datedate
+
amountdecimal
+
portfolio_code FKstring
+
counterparty_id FKbigint
+ +
" data-cls="whBroker" data-clsbase="drow "> +
broker_codestring
+
+
+
+
+ + additive +
+
+
+ +
+ ✓ reconciled — nothing lost +
+
+
+
+ + + \ No newline at end of file diff --git a/demos/standalone/Erwin.html b/demos/standalone/Erwin.html new file mode 100644 index 0000000..5c6860a --- /dev/null +++ b/demos/standalone/Erwin.html @@ -0,0 +1,169 @@ + + + + + +Erwin import + + + + +
+ +
+ ◮ + Modelith + import erwin + Bring an erwin model in. Own it in git. +
+ +
+ + +
+
+ + + + mdl — import +
+
+
# a model exported from erwin
+
fund.xml (erwin XML)
+
+
$ mdl import erwin fund.xml -o fund --scaffold model
+
scaffolded .mdl/lock.yaml
+
imported 3 entities, 1 relationships from erwin to fund
+
+
$ mdl validate -m fund
+
validation passed
+
+
# now it's plain YAML in git — yours to version, diff, and generate
+
$
+
+
+ + +
+
+ + +
+ <Entity Name="Counterparty">
+   <Attribute Name=
+     "Counterparty_Id" .../>
+   <Attribute Name=
+     "Legal_Name" .../>
+ </Entity>
+ <Relationship .../>
+ … +
+ + +
→
+ + + + + + + +
+
counterparty
+
+
🔑 counterparty_idid_type
+
legal_namestring
+
+
+ +
+
trade
+
+
🔑 trade_idstring
+
counterparty_id FKid_type
+
+
+ +
+
institution
+
+
counterparty_idstring
+
+
+
+
+
+
+ + + \ No newline at end of file diff --git a/demos/standalone/Main.html b/demos/standalone/Main.html new file mode 100644 index 0000000..a13aa3d --- /dev/null +++ b/demos/standalone/Main.html @@ -0,0 +1,202 @@ + + + + + +Design-first + + + + +
+ +
+ ◮ + Modelith + design-first + Design in git. Compile to contract-enforced dbt. +
+ +
+ + +
+
+ + + + mdl — shop +
+
+
$ mdl model render
+
# model: shop
+
entity customer · order · order_line · product
+
+
$ mdl new entity payment --definition "A settlement against an order"
+
created entity 'payment' (01M3WX64NZ…)
+
+
$ mdl validate
+
validation passed
+
+
$ mdl generate
+
wrote 9 files to target/dbt
+
+
$ mdl export contract
+
wrote contract to datacontract.yaml
+
$
+
+
+ + +
+
+ + + + + + + + +
+
customer
+
+
🔑 customer_idbigint
+
emailstring
+
full_namestring
+
created_attimestamp
+
+
+ + +
+
order
+
+
🔑 order_idbigint
+
customer_id FKbigint
+
order_datedate
+
statusstring
+
+
+ + +
+
order_line
+
+
🔑 order_line_idbigint
+
order_id FKbigint
+
product_id FKbigint
+
quantityinteger
+
+
+ + +
+
product
+
+
🔑 product_idbigint
+
skustring
+
namestring
+
unit_pricedecimal
+
+
+ + +
+
paymentnew
+
+
🔑 payment_idbigint
+
+
+ + +
✓ valid
+ + +
+
dbt · contract enforced
+
ODCS · data contract
+
+
+
+
+
+ + + \ No newline at end of file diff --git a/demos/standalone/Ontology.html b/demos/standalone/Ontology.html new file mode 100644 index 0000000..058a4d3 --- /dev/null +++ b/demos/standalone/Ontology.html @@ -0,0 +1,169 @@ + + + + + +Ontology anchoring + + + + +
+ +
+ ◮ + Modelith + ontology + Anchor the model to an industry vocabulary. +
+ +
+ + +
+
+ + + + mdl — ibor · FIBO +
+
+
# the model declares an industry vocabulary
+
ontology_stack:
+
- name: fibo-ols layer: industry type: ols
+
+
$ mdl ontology search counterparty
+
fibo:PartyInRole [fibo-ols]
+
A relationship that establishes the specific capacity
+
in which a party acts in some context.
+
+
$ mdl ontology align counterparty --to fibo:PartyInRole
+
aligned · recorded in counterparty.yaml
+
+
$ mdl ontology status
+
industry alignment coverage: 100.0%
+
$
+
+
+ + +
+
+ + +
+
FIBO · industry vocabulary
+
+
fibo:PartyInRole
+
The capacity in which a party acts in some context.
+
+
+
fibo:FinancialInstrument
+
A tradable monetary contract.
+
+
+
fibo:Position
+
A quantity of an instrument held by a party.
+
+
+ + + + + + + +
+
counterparty⚙ fibo
+
+
🔑 counterparty_idbigint
+
legal_namestring
+
countrystring
+
+
+ + +
+
industry alignment coverage
+
+
+
+
100%
+
+
+
+
+
+ + + \ No newline at end of file diff --git a/demos/standalone/build.mjs b/demos/standalone/build.mjs new file mode 100644 index 0000000..04bfb26 --- /dev/null +++ b/demos/standalone/build.mjs @@ -0,0 +1,101 @@ +// Convert the 6 .dc.html Design artboards into self-contained, animated HTML files that need +// no Design runtime, so they can be screen-recorded and embedded on the landing page. +// +// The .dc.html files bind {{holes}} to DCLogic.renderVals() stepped by setInterval. Holes are +// applied IN PLACE (never by rebuilding innerHTML, which would restart CSS transitions): each +// {{x}} in the template is turned into a marker the shim updates on the live element: +// class="base {{x}}" -> the element gets data-cls="x"; shim sets className = "base "+val +// style="…:{{x}};…" -> the element gets data-style-="x"; shim sets that style prop +// >{{x}}< (text) -> wrapped in ; shim sets textContent +import { readFileSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +const here = dirname(fileURLToPath(import.meta.url)); +const SRC = process.argv[2]; +if (!SRC) { console.error("usage: node build.mjs "); process.exit(1); } + +const files = ["Main.dc.html","Adopt.dc.html","Drift.dc.html","Ontology.dc.html","Erwin.dc.html","Compile.dc.html"]; + +for (const f of files) { + const raw = readFileSync(join(SRC, f), "utf8"); + const css = (raw.match(/[\s\S]*? + + +
${body}
+ + +`; + writeFileSync(join(here, f.replace(".dc.html",".html")), out); + console.log("wrote", f.replace(".dc.html",".html")); +} diff --git a/demos/standalone/package.json b/demos/standalone/package.json new file mode 100644 index 0000000..b0bd180 --- /dev/null +++ b/demos/standalone/package.json @@ -0,0 +1 @@ +{"type":"module","dependencies":{"puppeteer-core":"^21.11.0"}} \ No newline at end of file diff --git a/demos/standalone/record.mjs b/demos/standalone/record.mjs new file mode 100644 index 0000000..a0e57d2 --- /dev/null +++ b/demos/standalone/record.mjs @@ -0,0 +1,61 @@ +// Record each served standalone demo as an MP4 (+GIF) by driving headless Chrome over CDP, +// starting a screencast, and piping JPEG frames to ffmpeg at a fixed fps. +// Usage: node record.mjs +import puppeteer from "puppeteer-core"; +import { spawn } from "node:child_process"; +import { mkdirSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +const here = dirname(fileURLToPath(import.meta.url)); +const CHROME = process.argv[2]; +const PAGE = process.argv[3]; // e.g. Main.html +const SECONDS = Number(process.argv[4] || 16); +const FPS = 20; +const outDir = join(here, "out"); +mkdirSync(outDir, { recursive: true }); +const name = PAGE.replace(/\.html$/, ""); +const mp4 = join(outDir, name + ".mp4"); + +const browser = await puppeteer.launch({ + executablePath: CHROME, + headless: "new", + args: ["--no-sandbox", "--force-color-profile=srgb", "--hide-scrollbars", + "--window-size=1920,1080"], + defaultViewport: { width: 1920, height: 1080, deviceScaleFactor: 1 }, +}); +const page = await browser.newPage(); +await page.goto("http://localhost:8777/" + PAGE, { waitUntil: "networkidle0" }); +await new Promise(r => setTimeout(r, 1200)); // let fonts settle + 1st tick + +// Collect frames with their arrival time, then mux at a constant FPS by duplicating/dropping +// to the wall-clock timeline, so static stretches don't compress the video. +const frames = []; +const t0 = Date.now(); +const client = await page.target().createCDPSession(); +client.on("Page.screencastFrame", async ({ data, sessionId }) => { + frames.push({ t: Date.now() - t0, buf: Buffer.from(data, "base64") }); + try { await client.send("Page.screencastFrameAck", { sessionId }); } catch {} +}); +await client.send("Page.startScreencast", { format: "jpeg", quality: 92, everyNthFrame: 1 }); +await new Promise(r => setTimeout(r, SECONDS * 1000)); +await client.send("Page.stopScreencast"); + +// build a constant-FPS stream from the timestamped frames +const ff = spawn("ffmpeg", [ + "-y", "-f", "image2pipe", "-framerate", String(FPS), "-i", "pipe:0", + "-vf", "scale=1920:1080:flags=lanczos,format=yuv420p", + "-c:v", "libx264", "-preset", "medium", "-crf", "20", + "-movflags", "+faststart", mp4, +], { stdio: ["pipe", "inherit", "inherit"] }); +const total = SECONDS * 1000; +let fi = 0; +for (let k = 0; k < Math.floor(SECONDS * FPS); k++) { + const want = (k / FPS) * 1000; + while (fi + 1 < frames.length && frames[fi + 1].t <= want) fi++; + if (frames[fi]) ff.stdin.write(frames[fi].buf); +} +ff.stdin.end(); +await new Promise(res => ff.on("close", res)); +await browser.close(); +console.log("wrote", mp4); diff --git a/demos/tapes/_settings.tape b/demos/tapes/_settings.tape new file mode 100644 index 0000000..a3f2780 --- /dev/null +++ b/demos/tapes/_settings.tape @@ -0,0 +1,19 @@ +# Shared VHS settings for every Modelith demo — Source this at the top of each tape so the +# whole set is visually consistent (one font, theme, size, cadence). Tune here once. +# +# Why these values: 1280x900 is tall enough that a multi-command demo does not scroll the +# earlier commands off the top; 22px font stays legible when embedded small; the amber theme +# echoes the canvas accent; 70ms typing reads as a calm human typing (not a race); and each +# tape adds generous Sleeps after a command so the viewer can read the output before the next. + +Set Shell "zsh" +Set FontSize 22 +Set FontFamily "JetBrains Mono" +Set LetterSpacing 0 +Set LineHeight 1.2 +Set Width 1280 +Set Height 900 +Set Padding 28 +Set TypingSpeed 70ms +Set WaitTimeout 30s +Set Theme { "name": "Modelith", "black": "#14110f", "red": "#f0857a", "green": "#6cbf73", "yellow": "#e0a63a", "blue": "#7fb0d8", "magenta": "#d98ba8", "cyan": "#9fd0c7", "white": "#ece6dd", "brightBlack": "#4a3c2c", "brightRed": "#f0857a", "brightGreen": "#6cbf73", "brightYellow": "#f2c14e", "brightBlue": "#7fb0d8", "brightMagenta": "#d98ba8", "brightCyan": "#9fd0c7", "brightWhite": "#ffffff", "background": "#14110f", "foreground": "#ece6dd", "cursor": "#e0a63a", "selection": "#4a3c2c" } diff --git a/demos/tapes/films/1-design-first.tape b/demos/tapes/films/1-design-first.tape new file mode 100644 index 0000000..4be0e40 --- /dev/null +++ b/demos/tapes/films/1-design-first.tape @@ -0,0 +1,55 @@ +# FILM 1 — "Design-first": a new domain, from a blank repo to running dbt + a data contract. +# One continuous session — the project is set up once (hidden) and evolves on screen; the +# `clear`s are scene breaks, not resets. Every command and every line of output is real. +Source demos/tapes/_settings.tape +Output demos/out/1-design-first.mp4 + +Hide +Type "PROMPT='$ '" Enter +Type "setopt interactivecomments" Enter +Type "export PATH=/Users/dbose/projects/modelith/.venv/bin:$PATH" Enter +Type "rm -rf /tmp/mdl-film1 && cp -R /Users/dbose/projects/modelith/demos/fixtures/shop /tmp/mdl-film1 && cd /tmp/mdl-film1" Enter +Type "mdl --version >/dev/null 2>&1" Enter +Type "clear" Enter +Show +Sleep 1.2s + +# ── Scene 1: the model you've designed, in git as plain YAML ── +Type "# a data model, designed in git: entities, keys, relationships" Sleep 500ms Enter +Sleep 2.5s +Type "mdl model render | head -20" Sleep 700ms Enter +Sleep 7s +Type "clear" Sleep 300ms Enter +Sleep 800ms + +# ── Scene 2: evolve it — add an entity (ULIDs minted for you) ── +Type "# the model evolves: add an entity, Modelith mints its ULIDs" Sleep 500ms Enter +Sleep 2.5s +Type "mdl new entity payment --definition 'A settlement against an order'" Sleep 700ms Enter +Sleep 4.5s + +# every change is validated — schema, references, naming +Type "mdl validate" Sleep 700ms Enter +Sleep 5s +Type "clear" Sleep 300ms Enter +Sleep 800ms + +# ── Scene 3: compile the model to real, contract-enforced dbt ── +Type "# one model compiles to real dbt..." Sleep 500ms Enter +Sleep 2s +Type "mdl generate" Sleep 700ms Enter +Sleep 5s + +# the schema carries ENFORCED contracts, keyed back to the model by ULID +Type "awk '/- name: order$/{f=1} f{print} /primary_key/{if(f)exit}' target/dbt/models/schema.yml" Sleep 700ms Enter +Sleep 7s +Type "clear" Sleep 300ms Enter +Sleep 800ms + +# ── Scene 4: ...and to more than dbt — an ODCS data contract, from the same model ── +Type "# ...and to a data contract, from the same model" Sleep 500ms Enter +Sleep 2s +Type "mdl export contract" Sleep 700ms Enter +Sleep 4s +Type "head -16 datacontract.yaml" Sleep 700ms Enter +Sleep 7s diff --git a/demos/tapes/films/2-adopt-a-warehouse.tape b/demos/tapes/films/2-adopt-a-warehouse.tape new file mode 100644 index 0000000..f9922b3 --- /dev/null +++ b/demos/tapes/films/2-adopt-a-warehouse.tape @@ -0,0 +1,45 @@ +# FILM 2 — "Adopt a warehouse": take over a legacy dbt project with no data model. +# One continuous session; every command and every line of output is real. The reverse +# detection (SCD2, Data Vault, surrogate stripping) is the warehouse's real structure. +Source demos/tapes/_settings.tape +Output demos/out/2-adopt-a-warehouse.mp4 + +Hide +Type "PROMPT='$ '" Enter +Type "setopt interactivecomments" Enter +Type "export PATH=/Users/dbose/projects/modelith/.venv/bin:$PATH" Enter +Type "rm -rf /tmp/mdl-film2 && cp -R /Users/dbose/projects/modelith/demo/legacy-warehouse /tmp/mdl-film2 && cd /tmp/mdl-film2 && rm -rf owned" Enter +Type "mdl --version >/dev/null 2>&1" Enter +Type "clear" Enter +Show +Sleep 1.2s + +# ── Scene 1: where most teams actually are — a dbt warehouse, no model ── +Type "# an existing dbt warehouse: staging, marts, a Data Vault: but no data model" Sleep 500ms Enter +Sleep 2.5s +Type "ls models/staging models/marts models/vault" Sleep 700ms Enter +Sleep 6s +Type "clear" Sleep 300ms Enter +Sleep 800ms + +# ── Scene 2: one command lifts it into a model, patterns recognized ── +Type "# one command lifts it into a logical model" Sleep 500ms Enter +Sleep 2s +Type "mdl reverse --project target/manifest.json --out owned" Sleep 700ms Enter +Sleep 10s +Type "clear" Sleep 300ms Enter +Sleep 800ms + +# ── Scene 3: nothing is a black box — every inference is reviewable ── +Type "# every inference is a reviewable decision, not a guess" Sleep 500ms Enter +Sleep 2s +Type "mdl decisions list -m owned | head -12" Sleep 700ms Enter +Sleep 9s +Type "clear" Sleep 300ms Enter +Sleep 800ms + +# ── Scene 4: now you own it — it validates and reads as clean business entities ── +Type "mdl validate -m owned" Sleep 700ms Enter +Sleep 4.5s +Type "mdl model render -m owned | head -16" Sleep 700ms Enter +Sleep 8s diff --git a/demos/tapes/films/3-the-drift-gate.tape b/demos/tapes/films/3-the-drift-gate.tape new file mode 100644 index 0000000..4918b45 --- /dev/null +++ b/demos/tapes/films/3-the-drift-gate.tape @@ -0,0 +1,54 @@ +# FILM 3 — "The drift gate": the model and the warehouse stay in lockstep. A column lands in +# the warehouse; drift classifies it; reconcile folds the safe change into the model and the +# model is clean again. Every command, every edit, and every line of output is real — the +# drift and reconcile run against a genuine dbt manifest from `dbt parse`. +Source demos/tapes/_settings.tape +Output demos/out/3-the-drift-gate.mp4 + +Hide +Type "PROMPT='$ '" Enter +Type "setopt interactivecomments" Enter +Type "export PATH=/Users/dbose/projects/modelith/.venv/bin:$PATH" Enter +Type "rm -rf /tmp/mdl-film3 && cp -R /Users/dbose/projects/modelith/demo/ibor /tmp/mdl-film3 && cd /tmp/mdl-film3" Enter +Type "cp /Users/dbose/projects/modelith/demos/fixtures/add-broker-code.sh ." Enter +Type "MANIFEST=transform/warehouse/target/manifest.json" Enter +Type "mdl --version >/dev/null 2>&1" Enter +Type "clear" Enter +Show +Sleep 1.2s + +# ── Scene 1: the model and the warehouse agree ── +Type "# the committed model and the compiled dbt warehouse are in lockstep" Sleep 500ms Enter +Sleep 2.5s +Type "mdl drift -m model --manifest $MANIFEST" Sleep 700ms Enter +Sleep 5s +Type "clear" Sleep 300ms Enter +Sleep 800ms + +# ── Scene 2: a column lands in the warehouse (a real edit + a real dbt parse) ── +Type "# a teammate adds a 'broker_code' column to the transaction model + its contract" Sleep 500ms Enter +Sleep 3s +Type "bash add-broker-code.sh" Sleep 700ms Enter +Sleep 4s +Type "cd transform/warehouse && dbt parse -q && cd ../.." Sleep 700ms Enter +Sleep 5s +Type "clear" Sleep 300ms Enter +Sleep 800ms + +# ── Scene 3: drift catches it and classifies it — additive, not breaking ── +Type "# drift sees it, and classifies it: additive, safe" Sleep 500ms Enter +Sleep 2.5s +Type "mdl drift -m model --manifest $MANIFEST --check" Sleep 700ms Enter +Sleep 6s +Type "clear" Sleep 300ms Enter +Sleep 800ms + +# ── Scene 4: reconcile folds the safe change into the model — nothing lost ── +Type "# reconcile folds the safe change into the model" Sleep 500ms Enter +Sleep 2.5s +Type "mdl drift -m model --manifest $MANIFEST --reconcile" Sleep 700ms Enter +Sleep 6s + +# the model is in lockstep again, and still valid +Type "mdl drift -m model --manifest $MANIFEST && mdl validate -m model" Sleep 700ms Enter +Sleep 6s diff --git a/docs/assets/demos/Adopt.gif b/docs/assets/demos/Adopt.gif new file mode 100644 index 0000000..a4c428d Binary files /dev/null and b/docs/assets/demos/Adopt.gif differ diff --git a/docs/assets/demos/Compile.gif b/docs/assets/demos/Compile.gif new file mode 100644 index 0000000..9ffef59 Binary files /dev/null and b/docs/assets/demos/Compile.gif differ diff --git a/docs/assets/demos/Drift.gif b/docs/assets/demos/Drift.gif new file mode 100644 index 0000000..2c19a43 Binary files /dev/null and b/docs/assets/demos/Drift.gif differ diff --git a/docs/assets/demos/Erwin.gif b/docs/assets/demos/Erwin.gif new file mode 100644 index 0000000..c072f07 Binary files /dev/null and b/docs/assets/demos/Erwin.gif differ diff --git a/docs/assets/demos/Main.gif b/docs/assets/demos/Main.gif new file mode 100644 index 0000000..05ee50d Binary files /dev/null and b/docs/assets/demos/Main.gif differ diff --git a/docs/assets/demos/Ontology.gif b/docs/assets/demos/Ontology.gif new file mode 100644 index 0000000..9832800 Binary files /dev/null and b/docs/assets/demos/Ontology.gif differ diff --git a/uv.lock b/uv.lock index 30c80bd..9e9d8b1 100644 --- a/uv.lock +++ b/uv.lock @@ -939,7 +939,7 @@ requires-dist = [ [[package]] name = "modelith-dbt" -version = "0.6.12" +version = "0.6.13" source = { editable = "." } dependencies = [ { name = "dbt-artifacts-parser" },