diff --git a/CHANGELOG.md b/CHANGELOG.md index 141b50b..0a80a7a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ All notable changes to this project are documented in this file. ### Added - **The `--qc` study stage now fails the build on self-referential edges.** A new `self-loop-edges` assertion in `build-kg --qc`'s stage-7 study flags any edge whose `subject` and `object` are the same CURIE. In a table-derived graph that is almost always a resolution collapse (two differently worded cells landing on one identifier, or one cell wired into both statement slots), so the edge states nothing and every consumer has to filter it. The study reports the offending edge count with the repeated CURIE and its per-CURIE total (capped at 10 examples like the other assertions) and exits non-zero; it does not delete the edge, so the author sees which mapping collapsed and fixes it in the config. Only non-empty string pairs count: absent, null, and blank slots stay under `incomplete-edges`, and non-string slots are not flagged. Intentionally reflexive assertions fail a `--qc` build too (accepted caveat). The check streams the same single edge pass as the other assertions, so it adds no extra read. +- **`eq`/`ne` reindex filters can match cells with a Polars regex.** `source.reindex` entries gain `regex`, which defaults to `false` so every shipped config keeps its literal comparison; with `regex: true` an `eq`/`ne` entry treats `comparator` as a Polars-compatible pattern searched in the cell (`eq` keeps the matches, `ne` keeps the rest), so one anchored filter drops a whole missing-value family (`N/A`, `n/a`, `--`, `not reported`) instead of one literal filter per spelling. Matching is case-sensitive and unanchored, exactly like `exclude_regex`, cells are compared as text so a typed `parquet` column filters on its cast form, and a null cell matches nothing, so blank rows drop under either operator just as they do today. Config-time guards: `regex` with `lt`/`le`/`gt`/`ge` is rejected (`comparison-regex-requires-eq-ne`), and an uncompilable pattern, or one that can match an empty cell (so it matches every cell and an `ne` filter would silently empty the table), is rejected (`regex-bad-pattern`). The new `lib.reindex_regex` filter op carries the match, so a config that never names the key emits exactly the ops it emitted before and keeps its existing run-cache digests. ## 22.1.0 - 2026-10-22 diff --git a/docs/configuration/table.md b/docs/configuration/table.md index 4c87f12..41e670b 100644 --- a/docs/configuration/table.md +++ b/docs/configuration/table.md @@ -175,7 +175,8 @@ Filter rows based on column values. | ------- | ------ | ---------- | ------------- | | `column` | String | Yes | Source column letters to evaluate; constrained to `^[A-Z]{1,3}$` (`A`-`ZZZ`). | | `comparison` | String | No | Operator. Defaults to `"ne"`; allowed values are `"eq"`, `"ne"`, `"lt"`, `"le"`, `"gt"`, `"ge"`. | -| `comparator` | String\|Int\|Float | Yes | Value to compare against. Must be a string for `"eq"`/`"ne"`, or a number for `"lt"`/`"le"`/`"gt"`/`"ge"`. | +| `comparator` | String\|Int\|Float | Yes | Value to compare against. Must be a string for `"eq"`/`"ne"`, or a number for `"lt"`/`"le"`/`"gt"`/`"ge"`. With `regex: true` it is a pattern instead of a literal, so it must be a string. | +| `regex` | Boolean | No | Treat `comparator` as a Polars-compatible regular expression searched in the cell instead of a literal value: `"eq"` keeps rows whose cell matches, `"ne"` keeps rows that do not. Case-sensitive and unanchored (anchor with `^`/`$` to pin a whole cell), the cell is compared as text, and a null cell never matches, so blank rows drop under either operator. Only valid with `"eq"`/`"ne"` (`comparison-regex-requires-eq-ne`); a pattern that is invalid, whitespace-only, or can match an empty cell (`^`, `()`, `.*`) is rejected at config time (`regex-bad-pattern`) because such a pattern matches every cell, so an `"ne"` filter would silently drop the whole table. Defaults to `false` (literal comparison), so existing configs are unchanged. | **Example:** @@ -184,6 +185,21 @@ reindex: - {column: C, comparison: lt, comparator: 0.05} # Keep rows where column C < 0.05 ``` +**Regex comparators (`regex: true`).** One anchored pattern replaces a stack of literal `"ne"` filters for the same missing-value family: + +```yaml +reindex: + - {column: A, comparison: ne, comparator: '^(?:n/?a|-{1,2}|not reported)$', regex: true} # drop every placeholder spelling + - {column: B, comparison: eq, comparator: '^\d{4}$', regex: true} # keep only cells that are exactly four digits + - {column: C, comparison: ne, comparator: "N/A"} # literal comparison: `regex` defaults to false +``` + +- Matching is Polars `str.contains`, so it is **unanchored**: `n/?a` matches the `na` inside `bona fide` and drops that row. Write `^...$` to pin the whole cell. +- The dialect is the Rust `regex` crate, the same constraints documented under [Text Transformations](#text-transformations): no backreferences, no lookarounds, groups and character classes fine, and inline flags such as `(?i)` work. +- Cells are compared as text (the column is cast to String), so a typed `parquet` column matches its string form (`22.0` matches `^2\d?\.0$`). Use `"lt"`/`"le"`/`"gt"`/`"ge"` to compare numbers. +- A null cell matches nothing, so blank rows drop under both `"eq"` and `"ne"`, exactly as the literal `"eq"`/`"ne"` filters already drop them. +- Config-time guards: `regex: true` on an ordering comparison is rejected (`comparison-regex-requires-eq-ne`), and a pattern the Polars engine cannot compile, or one that can match an empty cell (`^`, `()`, `.*`, `x*`), is rejected (`regex-bad-pattern`) because both match every cell and an `"ne"` filter would then silently drop the whole table. + ### Statement (Triple Definition) Defines subject-predicate-object relationships. diff --git a/docs/examples.md b/docs/examples.md index 14b2d47..6c8a0a7 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -215,7 +215,7 @@ sections: ## Conditional Filtering with Reindex Filter a **CSV** of gene-disease associations down to significant, well-powered rows before building KGX -edges (reindex on column values). +edges (reindex on column values, with a Polars regex for the rows whose value has several spellings). **Data:** CSV with gene-disease associations and significance thresholds @@ -231,6 +231,7 @@ template: reindex: - {column: C, comparison: lt, comparator: 0.05} - {column: D, comparison: ge, comparator: 100} + - {column: A, comparison: ne, comparator: "^(?:n/?a|-{1,2}|unknown gene)$", regex: true} statement: subject: method: column @@ -258,6 +259,8 @@ template: - **Reindex filtering** keeps only rows where column C (p-value) < 0.05 AND column D (sample size) >= 100; multiple reindex conditions are ANDed together. - **Comparison operators**: `lt` (less than), `ge` (greater or equal), `eq`, `ne`, `gt`, `le`. +- **Regex comparator** (`regex: true` on an `eq`/`ne` entry) drops every placeholder gene cell in one filter (`N/A`, `n/a`, `--`, `unknown gene`) instead of one literal filter per spelling. The pattern is anchored with `^`/`$` because Polars matching is unanchored: unanchored `n/?a` would also drop `bona fide`. +- **Backwards compatibility**: `regex` defaults to false, so an `eq`/`ne` entry without it still compares the cell to the literal `comparator`. ---