Skip to content

Fix ontology lazy import - #45

Merged
dbose merged 3 commits into
mainfrom
fix-ontology-lazy-import
Sep 27, 2026
Merged

dbose merged 3 commits into
mainfrom
fix-ontology-lazy-import

Conversation

@dbose

@dbose dbose commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Fix Windows/core-install crashes in erwin import, and verify in a clean Docker install

Summary

A fresh Windows pip install modelith-dbt (core only, no [ontology] extra) crashed mdl import erwin in two different ways that our uv dev workspace could never reproduce. This PR fixes both, and adds an isolated Docker harness that exercises the flows in the install configuration users actually have, so this class of "green in dev, broken on install" bug is caught before shipping.

The bugs

  1. ModuleNotFoundError: No module named 'pyoxigraph' — mdl import erwin (and mdl init) into an empty folder scaffolds .mdl/lock.yaml via mdl_ontology.lock.Lock. Importing that submodule ran mdl_ontology/__init__.py, which eagerly imported the RDF-backed providers chain (pyoxigraph, the optional [ontology] extra). A core install doesn't have it, so scaffolding crashed — breaking the promise that the core CLI runs without the extra.

  2. OSError: [Errno 22] Invalid argument … \<root>.yaml — an erwin export carries a domain literally named <root>. The shared model writer (mdl_reverse.writer.write_model, used by every importer) used object names verbatim as filenames. < and > are illegal on Windows, so open() crashed. macOS/Linux silently wrote a <root>.yaml file, which is why local tests never caught it.

Traceback (most recent call last)
  File "C:\Users\XXXX\AppData\Roaming\Python\Python313\site-packages\mdl_cli\main.py", line 2956, in import_erwin_cmd
    write_reversed(result.model, write_target)

  File "C:\Users\XXXX\AppData\Roaming\Python\Python313\site-packages\mdl_reverse\writer.py", line 96, in write_model
    dump(f"logical/domains/{dom.name}.yaml", dom)

  File "C:\Users\XXXX\AppData\Roaming\Python\Python313\site-packages\mdl_reverse\writer.py", line 81, in dump
    path.write_text(dump_str(data), encoding="utf-8")

  File "C:\Program Files\Python313\Lib\pathlib\_local.py", line 555, in write_text
    return PathBase.write_text(self, data, encoding, errors, newline)

  File "C:\Program Files\Python313\Lib\pathlib\_abc.py", line 651, in write_text
    with self.open(mode='w', encoding=encoding, errors=errors, newline=newline) as f:

  File "C:\Program Files\Python313\Lib\pathlib\_local.py", line 537, in open
    return io.open(self, mode, buffering, encoding, errors, newline)

OSError: [Errno 22] Invalid argument:
'c:\\Users\\XXXX\\projects\\modelith-test\\model\\logical\\domains\\<root>.yaml'

The fixes

  • mdl_ontology/__init__.py now exports every public name lazily (PEP 562 __getattr__). Importing a pyoxigraph-free member (Lock) no longer pulls the backend; a backend-dependent name imports it only when accessed. _ontology() in the CLI force-resolves one backend-dependent export inside its guard, so a genuinely missing backend still fails with the modelith-dbt[ontology] install hint (exit 4), not a raw traceback.
  • mdl_reverse.writer now slugs every filename to a filesystem-safe form (strips < > : " / \ | ? *, control chars, trailing dots/spaces; never empty), and disambiguates collisions the slugging can create (Order Line + Order/Line → order_line.yaml + order_line-2.yaml). The object name is preserved in the model (a domain stays <root>); only the on-disk filename is sanitised. No names are dropped.

Verification (the durable part)

New sandbox/clean-install/ Docker harness: builds the wheel, installs it core-only (no extras) in python:3.12-slim, and runs the erwin import flow where a real user runs it — asserting the CLI loads, import into an empty folder scaffolds without the ontology extra, the <root> domain writes to a safe filename, and the model validates. ./sandbox/clean-install/run.sh does it in one command and exits non-zero on failure (pre-ship / CI gate). Verified passing on the current wheel, with a negative control confirming the container genuinely lacks pyoxigraph.

Tests

  • test_core_without_ontology.py: added init + import erwin cases under a blocked RDF backend (the scaffold path that crashed).
  • test_writer_filenames.py: illegal-char names → safe filenames + collision disambiguation.
  • Full suite: 919 passed, 1 skipped; ruff clean.

Notes

  • CLI modelith-dbt 0.6.8 → 0.6.10 (0.6.9 pyoxigraph, 0.6.10 filenames). VS Code extension unchanged at 0.3.16 — both fixes are CLI-side and the extension just shells out to mdl.
  • Windows box: install the built modelith_dbt-0.6.10-py3-none-any.whl.

On an install WITHOUT the [ontology] extra, `mdl init` and `mdl import
erwin` crashed with ModuleNotFoundError: No module named 'pyoxigraph'.
Scaffolding calls mdl_ontology.lock.Lock to write .mdl/lock.yaml, and
importing that submodule ran mdl_ontology/__init__.py, which eagerly
imported the RDF-backed providers/rdf_export chain (pyoxigraph). That
broke the core-install promise: the pyoxigraph-free Lock was unreachable.

mdl_ontology/__init__.py now exports every public name LAZILY (PEP 562
__getattr__), so importing a pyoxigraph-free member (Lock) no longer pulls
the backend; a backend-dependent name imports it only when accessed.

_ontology() in the CLI now force-resolves one backend-dependent export
(serialize) inside its try/except, so a missing backend still fails there
with the 'install modelith-dbt[ontology]' hint (exit 4) instead of leaking
a raw ImportError from the command body.

Regression tests: init + import erwin under a blocked RDF backend.

CLI 0.6.9.
An erwin export carries object names with characters that are illegal in
a Windows filename (< > : " / \ | ? *) — e.g. a domain named "<root>".
The writer used the name verbatim as a YAML filename, so on Windows the
write crashed with OSError [Errno 22] Invalid argument (it silently
produced a literally-named file on macOS/Linux, which is why tests missed
it). This is the shared serializer every importer (erwin, OSI, reverse)
uses, so the crash hit any Windows user with such a name.

- _slug() now strips/collapses Windows-illegal characters and trailing
  dots/spaces to underscores, and never returns empty. It was only
  lowercasing + replacing spaces before.
- Every filename in write_model now routes through _slug (domains,
  entities, relationships, subject-areas, physical tables previously used
  the raw or partially-cleaned name).
- A collision guard disambiguates names that slug to the same file
  ("Order Line" vs "Order/Line" -> order_line.yaml + order_line-2.yaml)
  so lossy slugging never silently overwrites one object with another.

The name itself is preserved in the model (a domain stays "<root>"); only
the on-disk filename is sanitised. No names are dropped.

CLI 0.6.10.
Our dev workspace (uv run) always has every optional dependency, so a
command that reaches the optional ontology stack passes locally and
crashes on a real `pip install modelith-dbt` without [ontology]. That is
how both recent Windows crashes shipped: import erwin pulled pyoxigraph
via the scaffold path, and a `<root>` domain produced a filesystem-illegal
filename — neither reproducible in the dev env.

Adds sandbox/clean-install/: a python:3.12-slim image that installs the
locally built wheel CORE-ONLY (no extras) and runs the erwin import flow
where a real user runs it — asserting the CLI loads, import erwin into an
empty folder scaffolds without the ontology extra, the <root> domain
writes to a safe filename, and the model validates. run.sh builds the
wheel, builds the image, and runs it in one command; exits non-zero on
failure so it is a pre-ship / CI gate. Verified: all checks pass on the
current wheel; the env genuinely lacks pyoxigraph (negative control).

Scoped to erwin import on a core install for now (the flows that broke);
init --workspace / config / generate bars and an [ontology] pass are the
natural extensions, plus a GitHub Actions job.
@dbose
dbose merged commit 18e876d into main Sep 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant