From 878455d39cee25830ed675c7f6cee7ed62322f77 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 10 Aug 2026 17:04:31 -0500 Subject: [PATCH 1/6] docs(specs): fold the diagram-traversal derivation into diagram.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Consolidate the expand/restrict traversal algebra (derived from first principles: the one primitive, edge rule R1, group rule R2, the additive expand and subtractive restrict, renamed-FK relabel fast-path, and the master-part group rule) into diagram.md as a 'Traversal algebra' design rationale — the single source of truth. cascade.md and trace.md now point to it as the downstream and upstream cases (expand down/up), keeping only their operation-specific content. Relocated from datajoint-python's DESIGN-expand-restrict.md (PR #1524), which is closed; the derivation and spec live with the other specs in datajoint-docs. --- src/reference/specs/cascade.md | 2 + src/reference/specs/diagram.md | 104 +++++++++++++++++++++++++++++++++ src/reference/specs/trace.md | 2 + 3 files changed, 108 insertions(+) diff --git a/src/reference/specs/cascade.md b/src/reference/specs/cascade.md index 8832567c..de5e9464 100644 --- a/src/reference/specs/cascade.md +++ b/src/reference/specs/cascade.md @@ -4,6 +4,8 @@ This document specifies how DataJoint propagates restrictions across the foreign For the user-facing entry points, see [Delete Data](../../how-to/delete-data.md). For dependent concepts, see [Master-Part](master-part.md), [Diagram](diagram.md), and [Data Manipulation](data-manipulation.md). +Cascade is the **downstream** case of the diagram-traversal algebra — the additive traversal `expand(direction="down")`. The rules it uses (the edge rule R1 and the group rule R2), and why they take the form they do, are derived once in the [Diagram spec's Design Rationale](diagram.md#traversal-algebra); this page specifies the cascade-specific behavior on top of them. + ## Overview A *cascade* starts at a (possibly restricted) **seed** table and propagates the restriction to every table that depends on it via foreign keys, so that a delete or preview affects all dependents consistently. Cascade is invoked by: diff --git a/src/reference/specs/diagram.md b/src/reference/specs/diagram.md index 89c94729..81bb7990 100644 --- a/src/reference/specs/diagram.md +++ b/src/reference/specs/diagram.md @@ -290,6 +290,110 @@ If a descendant table lives in a schema that hasn't been activated (loaded into --- +## Traversal algebra + +This design rationale derives the traversal operations from first principles. The operational methods above (`cascade`, `trace`, `restrict`) are not three separate features — they are a small, composable algebra over the dependency graph, derived here from first principles so the *rules*, not just the API, are the specification. The unifying model has **one additive traversal** (`expand`, of which `cascade` is the downstream case and `trace` the upstream case) and **one subtractive traversal** (`restrict`), built on **two rules** (an edge rule and a group rule). + +### 1. The one primitive: propagating a restriction across a foreign key + +A **restriction** on a table is a subset of its rows, written as a condition. + +Every foreign key `child -> parent` defines a function: each child row references exactly one parent row. Propagating a restriction across that edge is itself a **restriction** — restrict the neighbor by the restricted table, matched on the foreign-key attributes (`&` with a query expression). It works in either direction: + +- **downstream** (a restriction on the parent, carried to the child): `child & parent_restricted` — the child rows whose parent is in the restricted set. +- **upstream** (a restriction on the child, carried to the parent): `parent & child_restricted` — the parent rows referenced by the restricted child. + +Downstream and upstream are the *same* operation pointed opposite ways along the same foreign key. This is the whole engine; everything below is how the edge rule degenerates, what parts add, and how you accumulate across many edges. + +### 2. The edge rule (R1, referential) + +**Base case — the foreign key is the whole primary key, not renamed, no parts.** The parent's primary key is embedded verbatim in the child's, with the same column names. A primary-key restriction on the parent is a predicate on exactly those columns, which the child also has, by the same names. So: **carry the restriction unchanged** — the identical predicate that selects the parent rows already selects the matching child rows. This is why the same primary-key restriction rides the whole diagram. + +**Complication A — secondary foreign key.** The parent's key lands in the child's *secondary* (non-primary) attributes. A raw predicate still selects the right child rows, but the restriction is no longer a statement about the child's *identity*, so it cannot be promoted to the child's primary key and ridden further. Keep it relational: project the restricted parent to its key and restrict the child by it, matched on the foreign-key columns. + +**Complication B — renamed foreign key.** The referencing columns have different names in the child. The parent's predicate names columns the child lacks. Fix, mechanically: rename the restriction's columns through the foreign key's attribute map before restricting (reverse the rename going upstream). + +> **R1 (edge rule):** propagate a restriction across a foreign-key edge by **restricting** the neighbor by the restricted table (`&`), projected/renamed onto the shared foreign-key columns. When the foreign key is the whole primary key and unrenamed, the projection is the identity and the restriction collapses to "apply the same predicate." + +### 3. The group rule (R2, compositional) + +Part tables add **compositional** integrity on top of referential integrity: a master and its parts are one entity, created and deleted all-or-nothing. + +- **master -> part** needs nothing new — a part carries `-> master` in its primary key, so R1 already sweeps in all parts of a restricted master. +- **part -> master** is the new rule. A restriction landing on part rows satisfies referential integrity by touching just those rows, but leaves a fragment of an entity. So it must **lift existentially to the master** (the master is in if *any* of its parts is), and the master re-expands to **all** its parts. + +> **R2 (group rule):** a restriction touching any part of a master's group brings the whole group — existential lift part -> master, then expand master -> all parts. + +R2 is a closure over the master–part grouping, which is exactly why foreign-key restrictions alone cannot express it. + +### 4. Two operations over R1 + R2 + +There are exactly two irreducible ways to use the rules, and they are opposites. + +**`expand` — additive (grow from one seed).** A constructor: seed a single restricted table and grow outward by R1 + R2, accumulating reachable rows by **union** (a table is reached if reachable via any path). Directional, `direction="down" | "up" | "both"` (default `"down"`): + +- `direction="down"` — descendants: the **delete blast radius**. This is `cascade`. +- `direction="up"` — ancestors: the **valid query sources** a `make()` may read under the reproducibility contract. This is `trace`; inside `make()`, `self.upstream` is `expand(self & key, direction="up")`. +- `direction="both"` — a referentially-consistent **export region** around the seed. + +A single-seed additive closure is always consistent and never needs an intersection: tracing up pulls exactly the referenced ancestors, cascading down pulls exactly the dependents. + +**`restrict` — subtractive (progressively carve any diagram).** An instance method on any diagram, carving it down by applying conditions, accumulating by **intersection** — **every table is restricted by the conjunction of all conditions that reach it**; tables that go empty drop out. It is: + +- **progressive / chainable** — `.restrict(A).restrict(B)…`, each carves further; +- **order-independent** — the result is the conjunction of all conditions; +- **monotone** — every step only removes; every intermediate is a valid slice. + +`restrict` is *not* reducible to combinations of `expand`, because it applies **multiple independent conditions** and gives each table the AND of the ones upstream of it. Example: "all data for `subject_id=5` **and** `method_id=5`" — `Subject` and `ProcessingMethod` are independent ancestors meeting only at a shared descendant; combining single-seed expansions cannot assemble it, chained `restrict` does. + +### 5. Why this is the whole story + +- **Intersection is not a convergence rule.** It only arises in the subtractive model with multiple independent conditions (`restrict`). The additive model (`expand`) is pure reachability — always union. +- **The two compose freely.** A diagram is a set of tables, each holding one row-set. `expand` unions reachable rows in (grow); `restrict` intersects a propagated condition in (carve). One representation, so they chain in any order. A grow step ORs rows in (restrict by a list, `[cond, …]`); a carve step ANDs a further restriction on (chained `&`). +- **Materialization is a delete-time concern, not part of traversal.** Freezing a group's keys before deleting (delete runs parts-before-masters) matters only when a traversal feeds `delete`; the read-only closures never pay for it. + +### 6. Renamed foreign keys and the seed restriction + +Each time a restriction crosses a foreign key it must be re-expressed in the neighbor's attribute names. Renaming is the only thing that changes names across an edge, so it is the only place this needs care, and the *shape* of the seed restriction decides how. A restriction is one of three kinds: + +- **materialized** — a dict of primary-key values, or a sequence of them (literal `attr: value` rows, e.g. `A.keys()`); +- **subquery** — a query expression (another table, possibly restricted); +- **string** — a raw SQL predicate over attribute names, e.g. `'weight > 10'`. + +Only the materialized kind is *frozen literal values*; the other two are *live* (evaluated against current data). This split decides whether a renamed edge is crossed by simply relabelling or must be crossed relationally. + +**Kind 1 — materialized (relabel fast-path).** Crossing a renamed foreign key, the neighbor's restriction is obtained by **renaming the dict's keys through the edge, values unchanged**: keep the referenced attributes (relabelled) and drop any key field that does not exist on the neighbor. Going upstream this drops the child's own identity attributes, leaving exactly the parent's key; going downstream nothing is dropped and the child's own key attributes stay unconstrained (a partial key). Renamings chain, so a key relabels edge by edge. This is exact because the renaming is pure (values and types preserved) and attribute identity across the edge is fixed by the edge's pairing, not by coincidental name matches. A sequence of dicts (e.g. `A.keys()`) is the OR of its members and relabels element by element; the walk stays symbolic — no subqueries — and, being frozen literals, is stable and delete-safe. + +*Which attributes cross an edge:* an attribute propagates across an edge iff that edge **carries** it (the foreign key references it). A data attribute (`weight`) is carried by no edge and never propagates — a key containing one is really an `A & cond` case (Kind 2/3), reducible to Kind 1 by materializing to keys first, `(A & cond).keys()`. A *secondary* foreign-key attribute propagates along its own edge even though it is not part of the primary key. So the test is per edge — *does the key cover the attributes this edge carries?* — not a global "is the key primary-key-only?". + +*Direction, because a foreign key is a function:* **down** (parent to children) is the preimage — a parent key relabels to a partial child key and always suffices. **Up** (child to parent) is the image — to name the referenced parent by relabelling, the key must include the parent's **full primary key** (in the child's names). When it doesn't, `expand` (which must be exact) **materializes** — queries the child for the actual referenced parent keys — while `restrict` (which removes only the provably-excluded) may **leave the parent uncarved** (a looser but still referentially-valid slice). Same relabel condition for both; only the fallback differs. + +**Kinds 2 & 3 — live (restrict-then-project).** A subquery or string restriction has no literal values to relabel; cross the edge **relationally** — restrict `A` by the condition, project it onto the neighbor's referenced attributes (renamed), and restrict the neighbor. A string cannot cross as text (it may name attributes the foreign key doesn't carry, and rewriting SQL to the neighbor's names is not reliable); only its *effect* crosses, through the referenced-attribute values of the surviving `A` rows. Any live restriction becomes relabel-able (and delete-safe) the moment it is materialized to keys, `(A & r).keys()` — which is exactly what `cascade` does at plan time. + +### 7. The group rule and the relabel fast-path + +R2 needs **no new key machinery** — it is the relabel fast-path run twice, with the part-specific attribute deliberately lost in between. For a master `Session` (primary key `session_id`) and part `Session.Trial` (primary key `(session_id, trial_id)`): + +- **master to part (down):** the ordinary relabel — `Session & {'session_id': 5}` becomes the partial key `{'session_id': 5}` on `Session.Trial`, selecting *every* trial of session 5. "All parts follow the master" falls straight out of the down rule. +- **part to master (up), the existential lift:** relabel-drop — `Session.Trial & {'session_id': 5, 'trial_id': 2}` keeps `session_id`, drops the part-specific `trial_id`, giving `{'session_id': 5}`. A sequence of part keys across many trials all drop to the same master key and de-duplicate, so the OR-over-siblings is free. +- **master to all parts (re-expansion):** *not* a relabel of the seed — a fresh downstream step from the recovered master key, `{'session_id': 5}` on `Session` relabels down to `{'session_id': 5}` on `Session.Trial`, which drops the `trial_id` constraint and so *widens* from "trial 2" to all trials. + +**The signature: a key that reaches a part via its master carries no part-specific constraint.** The lift *narrows* the key to the master; the re-expansion *widens* it to all parts; the part-specific attribute is destroyed by the lift and cannot be recovered. That loss *is* compositional atomicity in key terms — the whole part-group comes along precisely because the returning key no longer distinguishes one part from its siblings. + +Corollaries: a materialized-key seed stays materialized through the whole part -> master -> parts round trip, so it is delete-safe for free (the delete-order materialization only ever fires for *live* seeds); and one mechanism serves all three call sites — the mutating part->master cascade, the upstream `trace` that surfaces an ancestor master's parts, and `restrict` promoting a part predicate to its master. + +### Summary + +| | additive (grow) | subtractive (carve) | +|---|---|---| +| operator | `expand(seed, direction)` (`cascade` = down, `trace` = up) | `diagram.restrict(*conditions, direction)` | +| accumulate | union | intersection | +| serves | delete blast radius / `make()` sources / export region | multi-condition pipeline carving | + +One data structure, two composable transforms, two rules — R1 (edge restriction) and R2 (group). `cascade` and `trace` are the downstream and upstream cases of the additive traversal. + +--- + ## Output Methods ### Graphviz Output diff --git a/src/reference/specs/trace.md b/src/reference/specs/trace.md index 8a9b78a5..5ca1cd4c 100644 --- a/src/reference/specs/trace.md +++ b/src/reference/specs/trace.md @@ -12,6 +12,8 @@ The two are designed as a unit. `trace` is the underlying graph operation; `self For the related downstream operation, see [Cascade Specification](cascade.md). For `make()` itself, see [AutoPopulate Specification](autopopulate.md). +Trace is the **upstream** case of the diagram-traversal algebra — the additive traversal `expand(direction="up")`, the mirror of cascade's downstream `expand(direction="down")`. The shared rules (the edge rule R1 and the group rule R2) are derived once in the [Diagram spec's Design Rationale](diagram.md#traversal-algebra); this page specifies the upstream-specific behavior and the `self.upstream` surface on top of them. + ## Why this exists A computed row's reproducibility rests on the convention that `make(self, key)` reads only from its declared upstream dependencies. Making that convention easy to follow — and its result easy to inspect — requires two pieces working together: From 3a824278e4c25745c6b0402e43225a9b456c24c5 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 10 Aug 2026 17:48:49 -0500 Subject: [PATCH 2/6] docs(specs): collapse the traversal algebra to a single expand operation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drop restrict and the additive/subtractive split from the diagram.md derivation. The model is now one operation, expand(seed, direction in {up,down,both}): cascade = down, trace = up, export = both, over the two rules R1 (edge) and R2 (group). A filter over several independent tables is not a second operation — it either seeds a common descendant (which inherits both keys) and expands both ways, or, lacking one, is a union of expansions; the per-table 'AND of upstream conditions' carving is a UI-layer composition of expands, not a core primitive. --- src/reference/specs/diagram.md | 38 ++++++++++++++-------------------- 1 file changed, 15 insertions(+), 23 deletions(-) diff --git a/src/reference/specs/diagram.md b/src/reference/specs/diagram.md index 81bb7990..0f0f6b2a 100644 --- a/src/reference/specs/diagram.md +++ b/src/reference/specs/diagram.md @@ -292,7 +292,7 @@ If a descendant table lives in a schema that hasn't been activated (loaded into ## Traversal algebra -This design rationale derives the traversal operations from first principles. The operational methods above (`cascade`, `trace`, `restrict`) are not three separate features — they are a small, composable algebra over the dependency graph, derived here from first principles so the *rules*, not just the API, are the specification. The unifying model has **one additive traversal** (`expand`, of which `cascade` is the downstream case and `trace` the upstream case) and **one subtractive traversal** (`restrict`), built on **two rules** (an edge rule and a group rule). +This design rationale derives the traversal operations from first principles. The operational methods above (`cascade` and `trace`) are not separate features — they are the two directions of **one traversal**, `expand`, over the dependency graph, derived here so the *rules*, not just the API, are the specification. The model is a single operation `expand(seed, direction)` — `cascade` is the downstream case, `trace` the upstream case — built on **two rules**: an edge rule (R1) and a group rule (R2). ### 1. The one primitive: propagating a restriction across a foreign key @@ -326,30 +326,22 @@ Part tables add **compositional** integrity on top of referential integrity: a m R2 is a closure over the master–part grouping, which is exactly why foreign-key restrictions alone cannot express it. -### 4. Two operations over R1 + R2 +### 4. The one operation: expand -There are exactly two irreducible ways to use the rules, and they are opposites. - -**`expand` — additive (grow from one seed).** A constructor: seed a single restricted table and grow outward by R1 + R2, accumulating reachable rows by **union** (a table is reached if reachable via any path). Directional, `direction="down" | "up" | "both"` (default `"down"`): +There is one traversal. Seed a single restricted table and grow outward by R1 + R2, accumulating reachable rows by **union** (a table is reached if reachable via any path). It is directional, `direction="down" | "up" | "both"` (default `"down"`): - `direction="down"` — descendants: the **delete blast radius**. This is `cascade`. - `direction="up"` — ancestors: the **valid query sources** a `make()` may read under the reproducibility contract. This is `trace`; inside `make()`, `self.upstream` is `expand(self & key, direction="up")`. -- `direction="both"` — a referentially-consistent **export region** around the seed. - -A single-seed additive closure is always consistent and never needs an intersection: tracing up pulls exactly the referenced ancestors, cascading down pulls exactly the dependents. - -**`restrict` — subtractive (progressively carve any diagram).** An instance method on any diagram, carving it down by applying conditions, accumulating by **intersection** — **every table is restricted by the conjunction of all conditions that reach it**; tables that go empty drop out. It is: +- `direction="both"` — a referentially-consistent **export region** around the seed: everything the seed rows depend on and everything derived from them. -- **progressive / chainable** — `.restrict(A).restrict(B)…`, each carves further; -- **order-independent** — the result is the conjunction of all conditions; -- **monotone** — every step only removes; every intermediate is a valid slice. +A single-seed closure is always consistent and never needs an intersection: tracing up pulls exactly the referenced ancestors, cascading down pulls exactly the dependents. Every requirement is a case of `expand`: blast radius (`down`), `make()` sources (`up`), and "all data for this entity" / consistent export (`both`). -`restrict` is *not* reducible to combinations of `expand`, because it applies **multiple independent conditions** and gives each table the AND of the ones upstream of it. Example: "all data for `subject_id=5` **and** `method_id=5`" — `Subject` and `ProcessingMethod` are independent ancestors meeting only at a shared descendant; combining single-seed expansions cannot assemble it, chained `restrict` does. +**Multiple conditions do not need a second operation.** A filter over several *independent* tables (e.g. "data for `subject_id=5` **and** `method_id=5`") is still `expand`. Either the conditions share a common descendant that inherits both keys — seed that descendant with the combined condition and `expand(both)` — or they do not, in which case no table is downstream of both, there is nothing to intersect, and the result is the union of the individual expansions. A per-table "conjunction of upstream conditions" carving is a UI-layer composition of `expand`s (a Navigator concern), not a core operation. ### 5. Why this is the whole story -- **Intersection is not a convergence rule.** It only arises in the subtractive model with multiple independent conditions (`restrict`). The additive model (`expand`) is pure reachability — always union. -- **The two compose freely.** A diagram is a set of tables, each holding one row-set. `expand` unions reachable rows in (grow); `restrict` intersects a propagated condition in (carve). One representation, so they chain in any order. A grow step ORs rows in (restrict by a list, `[cond, …]`); a carve step ANDs a further restriction on (chained `&`). +- **It is pure reachability.** A restriction reaches a table if it reaches it via *any* foreign-key path, so convergence is always **union** — there is no intersection to reason about and no second, subtractive operation. +- **One data structure.** A diagram is a set of tables, each holding one row-set; `expand` unions reachable rows into it as it grows. - **Materialization is a delete-time concern, not part of traversal.** Freezing a group's keys before deleting (delete runs parts-before-masters) matters only when a traversal feeds `delete`; the read-only closures never pay for it. ### 6. Renamed foreign keys and the seed restriction @@ -366,7 +358,7 @@ Only the materialized kind is *frozen literal values*; the other two are *live* *Which attributes cross an edge:* an attribute propagates across an edge iff that edge **carries** it (the foreign key references it). A data attribute (`weight`) is carried by no edge and never propagates — a key containing one is really an `A & cond` case (Kind 2/3), reducible to Kind 1 by materializing to keys first, `(A & cond).keys()`. A *secondary* foreign-key attribute propagates along its own edge even though it is not part of the primary key. So the test is per edge — *does the key cover the attributes this edge carries?* — not a global "is the key primary-key-only?". -*Direction, because a foreign key is a function:* **down** (parent to children) is the preimage — a parent key relabels to a partial child key and always suffices. **Up** (child to parent) is the image — to name the referenced parent by relabelling, the key must include the parent's **full primary key** (in the child's names). When it doesn't, `expand` (which must be exact) **materializes** — queries the child for the actual referenced parent keys — while `restrict` (which removes only the provably-excluded) may **leave the parent uncarved** (a looser but still referentially-valid slice). Same relabel condition for both; only the fallback differs. +*Direction, because a foreign key is a function:* **down** (parent to children) is the preimage — a parent key relabels to a partial child key and always suffices. **Up** (child to parent) is the image — to name the referenced parent by relabelling, the key must include the parent's **full primary key** (in the child's names). When it doesn't, the relabel fast-path cannot fire and `expand` **materializes** — queries the child for the actual referenced parent keys — and continues. **Kinds 2 & 3 — live (restrict-then-project).** A subquery or string restriction has no literal values to relabel; cross the edge **relationally** — restrict `A` by the condition, project it onto the neighbor's referenced attributes (renamed), and restrict the neighbor. A string cannot cross as text (it may name attributes the foreign key doesn't carry, and rewriting SQL to the neighbor's names is not reliable); only its *effect* crosses, through the referenced-attribute values of the surviving `A` rows. Any live restriction becomes relabel-able (and delete-safe) the moment it is materialized to keys, `(A & r).keys()` — which is exactly what `cascade` does at plan time. @@ -380,17 +372,17 @@ R2 needs **no new key machinery** — it is the relabel fast-path run twice, wit **The signature: a key that reaches a part via its master carries no part-specific constraint.** The lift *narrows* the key to the master; the re-expansion *widens* it to all parts; the part-specific attribute is destroyed by the lift and cannot be recovered. That loss *is* compositional atomicity in key terms — the whole part-group comes along precisely because the returning key no longer distinguishes one part from its siblings. -Corollaries: a materialized-key seed stays materialized through the whole part -> master -> parts round trip, so it is delete-safe for free (the delete-order materialization only ever fires for *live* seeds); and one mechanism serves all three call sites — the mutating part->master cascade, the upstream `trace` that surfaces an ancestor master's parts, and `restrict` promoting a part predicate to its master. +Corollaries: a materialized-key seed stays materialized through the whole part -> master -> parts round trip, so it is delete-safe for free (the delete-order materialization only ever fires for *live* seeds); and the same mechanism serves both directions — the mutating part->master cascade (`expand` down feeding a delete) and the upstream `trace` (`expand` up) that surfaces an ancestor master's parts. ### Summary -| | additive (grow) | subtractive (carve) | +| direction | operation | serves | |---|---|---| -| operator | `expand(seed, direction)` (`cascade` = down, `trace` = up) | `diagram.restrict(*conditions, direction)` | -| accumulate | union | intersection | -| serves | delete blast radius / `make()` sources / export region | multi-condition pipeline carving | +| `down` | `expand(seed, "down")` = `cascade` | delete blast radius | +| `up` | `expand(seed, "up")` = `trace` | `make()` sources (`self.upstream`) | +| `both` | `expand(seed, "both")` | consistent export region / "all data for this entity" | -One data structure, two composable transforms, two rules — R1 (edge restriction) and R2 (group). `cascade` and `trace` are the downstream and upstream cases of the additive traversal. +One operation, one data structure, two rules — R1 (edge restriction) and R2 (group). `cascade` and `trace` are the downstream and upstream cases of `expand`; a filter over several independent tables composes `expand`s rather than adding a new operation. --- From 9d90553ac2c09d0f0be6d4597f1d72e1364278c0 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 10 Aug 2026 17:51:14 -0500 Subject: [PATCH 3/6] docs(specs): align cascade/trace with the expand-only derivation cascade.md: DiGraph -> MultiDiGraph; the part-to-master walk enumerates all simple FK paths (not shortest_path) and defers the edge rule to diagram.md's R1. trace.md: recast the convergence table as the two directions of expand (cascade = down, trace = up), drop the restrict/AND row, and defer the rules to diagram.md (R1 + R2). Fixes statements that were stale after the MultiDiGraph migration and the expand-only model. --- src/reference/specs/cascade.md | 8 ++++---- src/reference/specs/trace.md | 19 +++++++++---------- 2 files changed, 13 insertions(+), 14 deletions(-) diff --git a/src/reference/specs/cascade.md b/src/reference/specs/cascade.md index de5e9464..ed910574 100644 --- a/src/reference/specs/cascade.md +++ b/src/reference/specs/cascade.md @@ -17,7 +17,7 @@ Both follow the same propagation rules; only the terminal step (delete vs. count ## Dependency graph -DataJoint loads its FK structure into a directed acyclic graph (`Connection.dependencies`, a `networkx.DiGraph`). The graph encodes two kinds of structure: +DataJoint loads its FK structure into a directed acyclic graph (`Connection.dependencies`, a `networkx.MultiDiGraph` — a multigraph so that parallel foreign keys, including renamed ones, are each their own edge rather than needing synthetic alias nodes). The graph encodes two kinds of structure: | Element | Encodes | |---|---| @@ -89,13 +89,13 @@ The Master is identified by **naming convention** via `dependencies.extract_mast ### Walking the FK path -The walk uses `nx.shortest_path(master, part)` to find the FK chain from Master to Part: +The walk enumerates **every** simple FK path from Master to Part (`nx.all_simple_edge_paths`), so a Part reachable through more than one foreign-key chain — or through parallel edges between the same pair — is restricted through all of them, combined with OR ([#1492](https://github.com/datajoint/datajoint-python/pull/1492)): ``` -Master → [intermediate Part(s)] → Part +Master -> [intermediate Part(s)] -> Part ``` -Each edge along the path fires one upward rule (U1, U2, or U3) per the edge's metadata (`attr_map`, `aliased`). **Intermediate Parts in a Part-of-Part chain are restricted along the way** — not only the Master. +Each edge along a path applies the edge rule R1 (see the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra)) in the upstream direction, per the edge's metadata (`attr_map`, renamed-or-not). **Intermediate Parts in a Part-of-Part chain are restricted along the way** — not only the Master. ### Materialization at the Master diff --git a/src/reference/specs/trace.md b/src/reference/specs/trace.md index 5ca1cd4c..75981bb3 100644 --- a/src/reference/specs/trace.md +++ b/src/reference/specs/trace.md @@ -27,21 +27,20 @@ Without (1), downstream tools that need row-level lineage (data-lineage viewers, ### Trace as the upstream mirror of cascade -`Diagram.cascade()` walks **downstream** from a restricted seed and answers *"what is affected if these rows are deleted?"* `Diagram.trace()` walks **upstream** and answers *"what contributed to these rows?"*. The two share the same dependency graph, the same edge model, and most of the same propagation machinery — only the direction differs. +`Diagram.cascade()` walks **downstream** from a restricted seed and answers *"what is affected if these rows are deleted?"* `Diagram.trace()` walks **upstream** and answers *"what contributed to these rows?"*. They are the two directions of the same traversal (`expand`), over the same dependency graph and the same rules — only the direction differs: -| Method | Direction | Convergence | Question answered | -|---|---|---|---| -| `Diagram.cascade(expr)` | downstream | OR — any FK path taints | What's affected if these rows are deleted? | -| `Diagram.restrict(expr)` | downstream | AND — must satisfy all FK paths | What satisfies all of these conditions? | -| `Diagram.trace(expr)` | **upstream** | **OR** — any FK path contributes | What contributed to these rows? | +| Method | `expand` direction | Question answered | +|---|---|---| +| `Diagram.cascade(expr)` | `down` | What's affected if these rows are deleted? | +| `Diagram.trace(expr)` | `up` | What contributed to these rows? | -`trace` uses OR convergence because an ancestor entity contributes to a child row if it appears via *any* FK path. (An AND-flavored upstream analog — "ancestors that contributed via *every* path" — is not provided in 2.3.) +Convergence is always **union** — an ancestor contributes to a child row if it is reachable via *any* foreign-key path. This is reachability, not an intersection; there is no AND-flavored upstream analog. -### Reusing the propagation primitives +### Reusing the propagation rules -`trace` applies the **upward propagation rules** (`U1`, `U2`, `U3`) defined in the [Cascade Specification](cascade.md#upward-propagation-child-parent), which are the symmetric inverses of `cascade`'s forward rules. Renamed FKs (`.proj()`) are reversed via U2; Part-of-Part chains are walked through naturally; multiple foreign keys between the same pair of tables are each reversed independently. +`trace` applies the **edge rule R1** in the upstream direction — the same rule `cascade` applies downstream — plus the **group rule R2** for master–part. Both are derived once in the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra); a renamed FK is relabelled through the edge's attribute map, Part-of-Part chains are walked naturally, and parallel foreign keys are each handled independently. -This is why `trace` cannot ship before the upward primitives exist in the codebase. As of DataJoint 2.3, the primitives are in place (added with [#1429's cascade fix](https://github.com/datajoint/datajoint-python/pull/1468)), and `trace` is a direct consumer. +This is why `trace` cannot ship before the upstream primitives exist in the codebase. As of DataJoint 2.3 they are in place (added with [#1429's cascade fix](https://github.com/datajoint/datajoint-python/pull/1468)), and `trace` is a direct consumer. ### The `make()` read/write boundary From afa8179d9ea32e509fc002d4d479118a0e21692c Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 10 Aug 2026 17:51:50 -0500 Subject: [PATCH 4/6] docs(cascade): drop the single-FK-path limitation (fixed by #1492 all-paths walk) --- src/reference/specs/cascade.md | 1 - 1 file changed, 1 deletion(-) diff --git a/src/reference/specs/cascade.md b/src/reference/specs/cascade.md index ed910574..0f461b0b 100644 --- a/src/reference/specs/cascade.md +++ b/src/reference/specs/cascade.md @@ -203,7 +203,6 @@ For a cascade subgraph with N nodes and E edges, propagation runs in at most O(N The following are known, documented behaviors of the cascade engine as shipped: -- **Single FK path (part→master walk).** The upward walk uses `nx.shortest_path` to find the FK chain from a Part to its Master. If a Part reaches its Master through multiple distinct FK chains, restrictions carried by the non-shortest paths are not applied. Workaround: use `part_integrity="ignore"` and perform the additional deletes manually. - **Materialization memory cost.** The master restriction is materialized via `to_arrays()` (required by the reverse-topological delete order — not merely MySQL error 1093; see [Materialization at the Master](#materialization-at-the-master)). The cost is bounded by the number of distinct master rows referenced by the matching parts. Cascade **preview** (`Diagram.cascade(...).counts()`) pays the same materialization cost as an actual delete. - **Empty-match sentinel.** When no master rows match, the master carries an always-false restriction and appears with zero rows in `counts()` and iteration. This is by design, not an error. - **Enforce granularity.** The `part_integrity="enforce"` post-check is **table-level**: it verifies that *some* rows of the master table were deleted whenever part rows were, not that each deleted part row's *specific* master row was deleted. As a result, rare false negatives (an unrelated master row happened to be deleted in the same cascade, masking a genuine orphan) and false positives (deleting already-orphaned part rows whose master was removed earlier) are possible. From 1019d95b64d1f034a99e1d4a300f2de32df11acd Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 10 Aug 2026 17:59:29 -0500 Subject: [PATCH 5/6] =?UTF-8?q?docs(specs):=20finish=20traversal=20de-dup?= =?UTF-8?q?=20=E2=80=94=20defer=20cascade/trace=20to=20R1/R2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit De-duplicate cascade.md's F1-3/U1-3 per-rule enumerations against the unified edge rule R1 in diagram.md#traversal-algebra, keeping the cascade-specific behavior on top. Rewrite worked examples to name R1's copy/rename/project cases instead of the old rule codes; drop the subset-symbol notation. Reframe part_integrity per the group rule R2: with cascade the master-part group is one item — master-first pulls in all parts, part-first lifts to the master then re-expands to all parts; ignore treats master and part as regular tables. Update trace.md's allowed-table-set contract so an ancestor master's whole part-group is in scope (R2 upstream), replacing the old non-descending note. Sweep residual unicode arrows to ASCII. --- src/reference/specs/cascade.md | 62 +++++++++++++--------------------- src/reference/specs/trace.md | 8 ++--- 2 files changed, 28 insertions(+), 42 deletions(-) diff --git a/src/reference/specs/cascade.md b/src/reference/specs/cascade.md index 0f461b0b..842e9a83 100644 --- a/src/reference/specs/cascade.md +++ b/src/reference/specs/cascade.md @@ -22,34 +22,20 @@ DataJoint loads its FK structure into a directed acyclic graph (`Connection.depe | Element | Encodes | |---|---| | **Node** | A table, named by its fully-qualified SQL identifier (e.g. ``` `schema`.`table_name` ```). The graph stores the table's primary key set as node data. | -| **Edge `parent → child`** | An FK constraint from `child` to `parent`. Edge data: `attr_map` (dict mapping child's FK columns → parent's referenced columns), `aliased` (true iff any column was renamed), `primary` (true iff the FK is in the child's primary key). | -| **Parallel edges** | A child can reference the same parent through more than one foreign key — for example two renamed (`.proj()`) references — so the same `parent → child` pair may be connected by multiple edges, each carrying its own `attr_map`/`aliased`/`primary`. | +| **Edge `parent -> child`** | An FK constraint from `child` to `parent`. Edge data: `attr_map` (dict mapping child's FK columns -> parent's referenced columns), `aliased` (true iff any column was renamed), `primary` (true iff the FK is in the child's primary key). | +| **Parallel edges** | A child can reference the same parent through more than one foreign key — for example two renamed (`.proj()`) references — so the same `parent -> child` pair may be connected by multiple edges, each carrying its own `attr_map`/`aliased`/`primary`. | The cascade engine operates on a copy of this graph (the `Diagram` class), recording per-table restrictions in `_cascade_restrictions` and the set of restricted attributes in `_restriction_attrs`. -## Restriction propagation rules +## Restriction propagation across an edge -When propagating a restriction across an edge `parent → child`, one of three rules applies. The rule depends on whether the FK renames columns (`aliased`) and on whether the parent's currently-restricted attributes (`parent_attrs`) are contained in the child's primary key (`child_pk`). +Cascade propagates a restriction across each foreign-key edge by the **edge rule R1** — derived in the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra) — restricting the neighbor by the restricted table, projected/renamed onto the shared foreign-key columns. R1 has three degenerate cases, keyed by whether the foreign key renames columns and whether the restricted attributes are the neighbor's whole primary key: -### Forward propagation (parent → child) +- **copy** — non-renamed, and the restricted attributes are the neighbor's primary-key attributes: carry the same predicate unchanged (the columns share names). +- **rename** — renamed foreign key: project the restricted table with its columns renamed through the edge's attribute map to the neighbor's names. +- **project** — non-renamed, but the restricted attributes are *not* (only) the neighbor's primary key: project the restricted table onto the shared foreign-key columns so the join matches on the right columns. -| Rule | Trigger | Effect on child | -|---|---|---| -| **F1. Copy** | `not aliased and parent_attrs` **non-empty** `and parent_attrs ⊆ child_pk` | Child inherits the parent's restriction directly (same attribute names; literal restriction values copy as-is). An empty attribute set takes rule 3. | -| **F2. Aliased rename** | `aliased` | Child's restriction is `parent.proj(**{fk_col: parent_col for fk_col, parent_col in attr_map.items()})` — the parent expression with columns renamed to match the child's column names. | -| **F3. Project** | `not aliased and parent_attrs ⊄ child_pk` | Child's restriction is `parent.proj()` — the parent projected to its primary key. | - -After applying the rule, the child's restricted-attributes set is updated to track what's now constrained on it. The child becomes a propagation source for its own children in the next pass. - -### Upward propagation (child → parent) - -Symmetric inverses of the forward rules. Used by `part_integrity="cascade"` to propagate a Part's restriction up to its Master through the FK chain. Edge metadata is the same; the direction of travel is reversed. - -| Rule | Trigger | Effect on parent | -|---|---|---| -| **U1. Copy** | `not aliased and child_attrs` **non-empty** `and child_attrs ⊆ parent_pk` | Parent inherits the child's restriction directly (shared attribute names). An empty attribute set takes rule 3. | -| **U2. Aliased reverse-rename** | `aliased` | Parent's restriction is `child.proj(**{parent_col: fk_col for fk_col, parent_col in attr_map.items()})` — the child expression with FK columns renamed back to the parent's column names. | -| **U3. Project** | `not aliased and child_attrs ⊄ parent_pk` | Parent's restriction is `child.proj(*attr_map.keys())` — the child projected onto its FK columns (which, when non-aliased, share names with the parent's PK) so the parent restriction joins on the right columns. | +Cascade applies R1 **forward** (parent to child) for the delete/preview walk. After each step the child's restricted-attribute set is updated, and the child becomes a propagation source for its own children in the next pass. For `part_integrity="cascade"`, R1 is also applied **upstream** (child to parent) to lift a Part's restriction to its Master — the same rule with tail and head swapped. ## Cascade flow @@ -74,14 +60,14 @@ Master-Part integrity reflects the contract that a Master row exists for every P | Mode | Behavior | |---|---| | `"enforce"` (default) | If a delete would remove a Part row without removing the corresponding Master row, the entire delete is rolled back with `DataJointError`. The Master row is checked **after** the delete; the integrity violation is detected post-hoc and reversed. The *intent* is row-level (each deleted Part row should have its Master deleted), but the shipped post-check is **table-level** — see [Limitations](#limitations) for the resulting rare false negatives and false positives. | -| `"ignore"` | No upward propagation; no post-check. Use when the Master row is intentionally preserved and the user has accepted that Part rows may be orphaned. Caller is responsible for the consequences. | -| `"cascade"` | **Upward propagation enabled.** When cascade reaches a Part, the Master is also restricted (via the upward rules below), and the Master then forward-cascades back down to **all** its Parts (siblings of the originating Part included). Used when the user wants the master-part group treated atomically. | +| `"ignore"` | No upward propagation; no post-check. Master and Part are treated as **regular tables** — each is restricted only through its own foreign keys, and a Part row may be left orphaned when its Master row is preserved. Caller is responsible for the consequences. | +| `"cascade"` | **The master-part group is treated as one item.** Whichever member the cascade encounters first, the whole group comes with it: if the **Master** is reached first, all of its Parts are included; if a **Part** is reached first, the restriction is lifted to its Master (via the upward walk below) and then all of the Master's Parts are included (siblings of the originating Part included). This is the group rule R2 (see the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra)) applied in either direction of travel. | This document focuses on the `"cascade"` mode; the `"enforce"` and `"ignore"` modes do not change the propagation graph. ## Part-to-Master upward propagation -When `part_integrity="cascade"` and the cascade reaches (or starts at) a Part node, the engine triggers an **upward walk** of the FK graph from the Part to its Master, applying the upward rules (U1, U2, U3) at each edge. +When `part_integrity="cascade"` and the cascade reaches (or starts at) a Part node, the engine triggers an **upward walk** of the FK graph from the Part to its Master, applying the edge rule R1 upstream at each edge. Reaching the Master first needs no walk — its Parts are pulled in by ordinary forward propagation. ### Identifying the Master @@ -141,19 +127,19 @@ class Subject(dj.Manual): """ ``` -`Recording`'s columns are `{src_subject, src_session, recording_id}` — none of them are named `subject_id`. The FK from `Subject.Session → Subject.Recording` is aliased. +`Recording`'s columns are `{src_subject, src_session, recording_id}` — none of them are named `subject_id`. The FK from `Subject.Session -> Subject.Recording` is aliased. When `(Subject.Recording & {"recording_id": 5}).delete(part_integrity="cascade")` runs: 1. **Seed-is-Part check.** `extract_master(Recording) == Subject`. Trigger the upward walk. -2. **FK path.** `shortest_path(Subject, Recording) = [Subject, Subject.Session, Subject.Recording]`. -3. **Walk reversed.** - - Edge `Subject.Session → Subject.Recording`: `aliased=True`. Apply **U2** — `Subject.Session` is restricted by `Subject.Recording.proj(subject_id='src_subject', session_id='src_session')`. - - Edge `Subject → Subject.Session`: `aliased=False`, `child_attrs={subject_id, session_id} ⊆ parent_pk={subject_id}`? No (`session_id` not in parent pk). Apply **U3** — `Subject` is restricted by `Subject.Session.proj(*attr_map.keys())`, projecting the child onto its FK columns; for this primary FK those columns are just `subject_id`, so this is equivalent to `Subject.Session.proj()` projected to `subject_id`. +2. **FK path.** The simple FK path from Master to Part is `[Subject, Subject.Session, Subject.Recording]` (`nx.all_simple_edge_paths`). +3. **Walk reversed** (R1 upstream at each edge). + - Edge `Subject.Session -> Subject.Recording`: renamed FK — the **rename** case. `Subject.Session` is restricted by `Subject.Recording.proj(subject_id='src_subject', session_id='src_session')`. + - Edge `Subject -> Subject.Session`: not renamed, but the child's restricted attributes `{subject_id, session_id}` are not (only) the parent's primary key `{subject_id}` (`session_id` is not in it) — the **project** case. `Subject` is restricted by `Subject.Session` projected onto its foreign-key columns, which for this primary FK is just `subject_id`. 4. **Materialize Master.** `Subject`'s restriction is fetched into a value tuple; replaces the chained `QueryExpression`. 5. **Forward pass.** Master forward-cascades back down to `Subject.Session` and `Subject.Recording` (and any sibling Parts not on the original path), now with the materialized restriction. -Without the FK walk (the pre-fix behavior), the engine joined `subject_ft.proj() & recording_ft.proj()` on shared attribute names. `Subject` has `subject_id`; `Recording` has `src_subject`. No shared columns → empty restriction → Master not restricted. This is the failure mode from [#1429](https://github.com/datajoint/datajoint-python/issues/1429) Case 1. +Without the FK walk (the pre-fix behavior), the engine joined `subject_ft.proj() & recording_ft.proj()` on shared attribute names. `Subject` has `subject_id`; `Recording` has `src_subject`. No shared columns -> empty restriction -> Master not restricted. This is the failure mode from [#1429](https://github.com/datajoint/datajoint-python/issues/1429) Case 1. ### Example 2: Part-of-Part with no Master reference in PartB @@ -177,15 +163,15 @@ class Master(dj.Manual): """ ``` -`PartB`'s definition references `Master.PartA`, not `master` directly. The FK chain Master → PartA → PartB still exists in the dependency graph (PartA's FK to Master is `aliased=False`; PartA → PartB is also `aliased=False`). +`PartB`'s definition references `Master.PartA`, not `master` directly. The FK chain Master -> PartA -> PartB still exists in the dependency graph (PartA's FK to Master is `aliased=False`; PartA -> PartB is also `aliased=False`). For `(Master.PartB & {"master_id": 1}).delete(part_integrity="cascade")`: -1. Upward walk: PartB → PartA → Master. - - Edge PartA → PartB: `aliased=False`, `child_attrs ⊆ parent_pk`? `child_attrs = {master_id}`, `parent_pk = {master_id, part_a_id}`. Yes ⊆. Apply **U1** — `PartA` inherits `PartB`'s restriction directly. - - Edge Master → PartA: `aliased=False`, `child_attrs = {master_id} ⊆ parent_pk = {master_id}`. Apply **U1** — `Master` inherits `PartA`'s restriction. +1. Upward walk: PartB -> PartA -> Master (R1 upstream at each edge). + - Edge PartA -> PartB: not renamed, and the child's restricted attributes `{master_id}` are within the parent's primary key `{master_id, part_a_id}` — the **copy** case. `PartA` inherits `PartB`'s restriction directly. + - Edge Master -> PartA: not renamed, and `{master_id}` is the parent's primary key `{master_id}` — the **copy** case. `Master` inherits `PartA`'s restriction. 2. Materialize Master. -3. Forward cascade Master → PartA → PartB picks up all sibling rows under `master_id=1`. +3. Forward cascade Master -> PartA -> PartB picks up all sibling rows under `master_id=1`. Without the FK walk (the pre-fix behavior), the engine jumped directly from PartB to Master via `master_ft.proj() & partB_ft.proj()`. PartA was never restricted, and the chain semantics were silently incorrect. This is [#1429](https://github.com/datajoint/datajoint-python/issues/1429) Case 2. @@ -195,8 +181,8 @@ For a cascade subgraph with N nodes and E edges, propagation runs in at most O(N ## What is not part of this specification -- **`Diagram.trace()`** for general upstream restriction propagation: a related but distinct feature that **shipped in 2.3** and reuses the same upward rules (U1/U2/U3) defined above. `trace()` exposes upstream propagation as a first-class operator; the cascade engine's upward walk in this document is the same machinery applied inside `part_integrity="cascade"`. See the [Upstream Trace Specification](trace.md) for `trace`'s API and semantics. -- **Custom propagation rules** (user-defined): not supported. The three forward and three upward rules cover the cases the FK graph can produce. +- **`Diagram.trace()`** for general upstream restriction propagation: a related but distinct feature that **shipped in 2.3** and applies the same edge rule R1 upstream. `trace()` exposes upstream propagation as a first-class operator; the cascade engine's upward walk in this document is the same machinery applied inside `part_integrity="cascade"`. See the [Upstream Trace Specification](trace.md) for `trace`'s API and semantics. +- **Custom propagation rules** (user-defined): not supported. R1 and its three cases (copy, rename, project) cover the cases the FK graph can produce. - **Cross-schema cascade**: handled by `dependencies.load_all_downstream()` called from `Diagram.cascade()`; orthogonal to the propagation rules described here. ## Limitations diff --git a/src/reference/specs/trace.md b/src/reference/specs/trace.md index 75981bb3..bbfc1389 100644 --- a/src/reference/specs/trace.md +++ b/src/reference/specs/trace.md @@ -78,7 +78,7 @@ Returns a `Diagram` instance whose nodes are the seed and all of its ancestors ( `trace` mirrors `cascade`: 1. Load the dependency graph via `connection.dependencies.load_all_upstream()` — the upstream analog of `load_all_downstream`, introduced with `Diagram.trace` ([#1423](https://github.com/datajoint/datajoint-python/issues/1423)). It discovers all schemas reachable via reverse FK edges from the seed's schema. -2. Take the seed's restriction and propagate it **upstream** along the FK graph. For each edge `parent → child`, when the child has a restriction, apply the upward rule (`U1`, `U2`, or `U3` per the cascade spec) to derive the parent's restriction. +2. Take the seed's restriction and propagate it **upstream** along the FK graph. For each edge `parent -> child`, when the child has a restriction, apply the edge rule R1 upstream (its **copy**, **rename**, or **project** case — see the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra)) to derive the parent's restriction. 3. Trim the resulting graph to **seed + ancestors only**. Descendants of the seed and unrelated ancestors are not included. 4. Convergence is **OR**: an ancestor entity is included if reachable through *any* FK path from the seed (consistent with how a child row "comes from" any of its FK parents). @@ -176,7 +176,7 @@ trace.counts() # '`imaging`.`__extract_traces`': 1, '`imaging`.`__summary`': 1} ``` -For a renamed-FK case (paralleling [Cascade Spec §Worked Example 1](cascade.md#example-1-part-of-part-with-renamed-fk)), the upward rules reverse the rename so `trace[Ancestor]` returns the ancestor with its native column names regardless of how the seed's columns are named. +For a renamed-FK case (paralleling [Cascade Spec §Worked Example 1](cascade.md#example-1-part-of-part-with-renamed-fk)), R1's **rename** case reverses the rename so `trace[Ancestor]` returns the ancestor with its native column names regardless of how the seed's columns are named. ## 2. `self.upstream` inside `make()` @@ -195,7 +195,7 @@ Once built, the trace diagram is reused for the remainder of the call. The under `self.upstream` exposes: - All declared ancestors of `self` (transitively, including renamed-FK chains). -- The Parts of ancestors that themselves lie on an FK path to `self` — a Part is included only when it is genuinely reachable through the FK graph, not merely because its master is an ancestor. +- **All Parts of every ancestor Master.** By the group rule R2 (see the [Diagram spec's Traversal algebra](diagram.md#traversal-algebra)), a master-part group is one item: once an ancestor Master is in the trace, its whole part-group comes with it — not only those Parts that happen to lie on an FK path to `self`. This is the upstream mirror of `part_integrity="cascade"` treating the group atomically. Requesting any table outside this set raises `DataJointError` — including tables that exist in the schema but are not ancestors of `self`. This is the same guarantee `Diagram.trace(...)` provides; `self.upstream` is just the per-`make()` instance of it. @@ -296,7 +296,7 @@ Teams adopt the read surface incrementally: - Source (shipped in 2.3): `src/datajoint/diagram.py` (`Diagram.trace`), `src/datajoint/autopopulate.py` (`AutoPopulate.upstream`). Implemented in [#1471](https://github.com/datajoint/datajoint-python/pull/1471) (trace) and [#1473](https://github.com/datajoint/datajoint-python/pull/1473) (self.upstream), building on the cascade rules from [#1468](https://github.com/datajoint/datajoint-python/pull/1468). - Issues: [#1423](https://github.com/datajoint/datajoint-python/issues/1423) (Diagram.trace), [#1424](https://github.com/datajoint/datajoint-python/issues/1424) (self.upstream). -- [Cascade Specification](cascade.md) — propagation rules (F1/F2/F3 forward, U1/U2/U3 upward) shared with `trace`. +- [Cascade Specification](cascade.md) — the downstream case of the same edge rule R1 and group rule R2 shared with `trace`. - [AutoPopulate Specification](autopopulate.md) — `make()` execution model and the make() reproducibility contract. - [Diagram Specification](diagram.md) — graph operations on the dependency graph. - [Entity Integrity](../../explanation/entity-integrity.md) — schema dimensions and FK semantics. From e54a70d58f06a9d08c8ebd72237fadba1a24ae12 Mon Sep 17 00:00:00 2001 From: Dimitri Yatsenko Date: Mon, 10 Aug 2026 18:00:31 -0500 Subject: [PATCH 6/6] docs(cascade): document delete-time materialization as a general rule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Downward expansion for delete cannot leave a restriction referencing a downstream table: reverse-topological delete empties it first, so the subquery matches zero rows and strands what should be deleted (#1496). State this as a general Delete-time materialization section — seed-referencing-descendant and the part-to-master master-key freeze are its instances — applied on every backend by delete-order reason, not as a MySQL-only (error 1093) concern. Preview/counts pays none of it. --- src/reference/specs/cascade.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/src/reference/specs/cascade.md b/src/reference/specs/cascade.md index 842e9a83..b8f83740 100644 --- a/src/reference/specs/cascade.md +++ b/src/reference/specs/cascade.md @@ -53,6 +53,16 @@ flowchart TB The engine performs **multiple passes** when `part_integrity="cascade"` is in effect: each pass forward-propagates over all `allowed_nodes`, and a pass may pull in a new master (with its descendants) that requires another pass. The loop terminates because the graph is a DAG and only finitely many nodes can be added. +## Delete-time materialization + +Downward expansion for **delete** carries a constraint that preview (count) does not: a table's restriction must not reference a table that is deleted *earlier* in the plan. + +`Table.delete` executes its per-table deletes in **reverse-topological order** (leaves first — see [Data Manipulation](data-manipulation.md)), so every downstream table is emptied before the table it depends on. If a table's restriction were left as a `QueryExpression` referencing a downstream table — which happens whenever the cascade **seed** is itself restricted by a descendant ([#1496](https://github.com/datajoint/datajoint-python/issues/1496)), or when the master's restriction is left pointing at an already-walked part — that subquery would run against a table already emptied by an earlier step, match zero rows, and silently strand the rows that should have been deleted. + +The fix is to **materialize** any such restriction to a literal key set at plan time, before any row is deleted, while the referenced table still holds its rows. Materialization is applied by delete-order reason on **every backend**, not made backend-conditional: the reverse-topological ordering strands rows regardless of engine. (MySQL additionally rejects a DELETE whose subquery targets the table being modified — error 1093 — but PostgreSQL, which permits that self-reference, still needs materialization for the ordering reason, so this is not a MySQL-only concern.) Where materialization is refused, the engine fails closed with a legible error rather than deleting through a stale subquery. Preview/`counts()` issues no deletes and so pays none of this cost. + +The master-key materialization in [Part-to-Master upward propagation](#materialization-at-the-master) below is the specific instance of this rule for the `part_integrity="cascade"` walk. + ## `part_integrity` modes Master-Part integrity reflects the contract that a Master row exists for every Part row that references it. Three modes govern how cascade enforces or relaxes that contract: @@ -87,9 +97,7 @@ Each edge along a path applies the edge rule R1 (see the [Diagram spec's Travers After the upward walk completes, the Master's accumulated restrictions are **materialized** to a literal value tuple via `(master_ft & restrictions).proj().to_arrays()` and stored as a single condition. Subsequent forward propagation from the Master back down to its other Parts then generates `WHERE pk IN (literal-list)` rather than `WHERE pk IN (SELECT ... FROM )`. -**Why materialization matters.** This is required for correctness on **every backend**, not merely to satisfy MySQL. `Table.delete` executes per-table deletes in reverse-topological order (leaves first — see [Data Manipulation](data-manipulation.md)), so the originating Part is deleted *before* the Master. If the Master's restriction were left as a `QueryExpression` referencing that Part, the Master's own DELETE — issued last — would find the Part already emptied, match zero rows, and silently strand the Master (the very compositional-integrity violation the upward walk exists to prevent). Materializing the Master's primary keys to a literal value set at plan time, before any rows are deleted, captures them while the Part still exists. - -A secondary consequence: the literal set also avoids a self-referential subquery. Left as a query, the Master forward-cascading back to the originating Part would generate a DELETE whose subquery targets the table being modified — which MySQL rejects ("error 1093: You can't specify target table 'T' for update in FROM clause"). PostgreSQL permits that self-reference, but the reverse-topological ordering above means materialization is required on **both** backends regardless — so this must not be treated as a MySQL-only concern. +This is the [delete-time materialization](#delete-time-materialization) rule applied here: the originating Part is deleted before the Master (reverse-topological order), so a Master restriction left pointing at that Part would match zero rows and strand the Master — the very compositional-integrity violation the upward walk exists to prevent. Capturing the Master's primary keys while the Part still holds its rows avoids that, and as a bonus avoids the self-referential subquery MySQL rejects with error 1093. Intermediate Parts in the chain are **not** materialized — they appear only as restrictions on the path, not as forward-cascade sources, so the self-reference issue doesn't arise there.