[SkyeAv]

feat: reindex rows by identifier membership with join and antijoin

Adds join and antijoin to the reindex comparison set so one config entry keeps or drops rows by exact membership in a list of identifiers, backed by polars semi/anti hash joins. Before this, "keep exactly these 40k CURIEs" needed one eq entry per identifier, and every one of those conditions was a separate full pass over the column; the new path replaces the whole list with one filter and beats the obvious is_in implementation at every list size worth writing down.

Config surface

• New operators: Comparisons.JOIN and Comparisons.ANTIJOIN (src/tablassert/enums.py), and Reindex.comparator widens to str | int | float | list[str] (src/tablassert/models.py).
• Coded rejects: comparison-membership-comparator-required (scalar comparator on a membership entry), comparison-membership-empty (an empty list would silently drop or keep the whole table), reindex-membership-column-missing (entry targets a column the source does not have), and comparison-membership-member-not-a-string.
• The last one is the reviewer-facing detail: comparator: [3, 4] is the natural spelling against an Int64 column whose string form is 3. Left to the field union it surfaced as a bare pydantic string_type error with no code and no docs URL, unlike the rest of the family, so a mode="before" validator now names the member and tells the user to quote it.
• Scalar behavior is untouched: the six existing comparisons still refuse a list, regex stays an eq/ne feature, and regex: true on a membership entry is rejected (comparison-regex-requires-eq-ne).

Runtime

• lib.reindex_membership(df, col, keys, anti=False): deduplicates the list, builds a one-column key frame, and runs how="semi" (keep) or how="anti" (drop).
• maintain_order="left" is explicit, not incidental: polars documents default join order as unspecified ("might differ across Polars versions or even between different runs") and the dependency is an open range, so a future 2.x must not be able to reshuffle a section's rows. Measured free (1M rows / 100k keys, median of five): semi 5.1ms -> 4.6ms, anti 5.4ms -> 5.4ms.
• One plan-time dispatcher: lib.reindex_op(entry) picks the op for a config entry. Scalar entries still emit exactly the (lib.reindex, (col, op, comp, cast)) tuples they always did, so existing run-cache digests do not move. Membership emits (lib.reindex_membership, (col, keys, anti)) and regex: true emits (lib.reindex_regex, (col, pattern, negate)); the dispatcher is shared with the regex work now on main (merged at 9fbb169).
• Typed columns: a textual column joins straight on col. Anything else is compared in the form polars renders (3, 3.0, true, 2024-01-15, 1.50 for a scale-2 decimal, 2024-01-15 06:24:30.000000+00:00 for a timezone-aware datetime), through a throwaway helper key column that is dropped before the op returns and is renamed out of the way if the source already carries that name (_membership_key_column).
• Refused dtypes: List, Array, Struct, Object, and Binary raise reindex-membership-column-dtype-unsupported rather than an opaque engine error or a filter that quietly matches nothing. Binary is refused because it casts only when the bytes happen to be valid UTF-8, so one list would match some rows and raise on the rest.
• Null semantics are join semantics: a null cell never matches, so join drops it while antijoin keeps it. That is the one place membership deliberately differs from ne, and it is what makes a deny-list section keep blank rows.
• Caching: registered as a filter phase with cost 1 in src/tablassert/runcache.py, so membership sections digest and reuse like the other filters; tests/test_run_cache.py pins that the identifier list is semantic (a changed list is a different cache entry) and that legacy scalar digests are unchanged.

Performance

tests/bench_reindex_membership.py, 1M string rows, every number the median of three timed windows (each window includes the key-frame build and the eager collect, which is what a build actually pays). This box, polars 2.0.0-rc.2:

keys	join	antijoin	is_in	~is_in
10	0.0035s	0.0052s	0.0037s	0.0038s
1,000	0.0031s	0.0043s	0.0038s	0.0050s
100,000	0.0170s	0.0227s	0.0253s	0.0275s
100,000 over Int64 (cast path)	0.0197s	-	-	-

• The claim is the ranking, not the seconds: the join ties the literal filter at ten keys and pulls about 1.5x ahead at 100k. Absolute numbers move with box load (a concurrent heavy job inflated every number by roughly 3x without changing the ranking).
• No-regression line: the unchanged scalar op measures lt 0.0011s and ne 0.0015s here, matching what it measured before the branch. The diff changes only how an entry compiles at plan time, never what the scalar filter does per row.
• Opt-in harness: all three tests skip unless TABLASSERT_BENCH=1, so default collection and CI never time anything.
• Enforced invariant in the harness: the join path may cost at most 2x the literal filter's median (asserted from 1k keys up, where the signal exists). It is deliberately not asserted to be strictly faster, because at ten keys the two tie inside noise.

Coverage

• tests/test_reindex_membership.py (new): keep/drop shapes, source order under a shuffled corpus, schema and dtype preservation, the null asymmetry, duplicate keys, empty/missing/non-list comparators, every refused dtype, the helper-column collision, the String fast path asserted from explain() (no helper column on a textual column, one on an Int64 column), op compilation for scalar/membership/regex entries, and the end-to-end section build.
• tests/test_models.py and tests/test_enums.py: the widened field, the membership validation family, and the new codes (also wired into tests/test_progress.py, which asserts each code surfaces with its docs URL through flatten_pydantic_error).

Docs

• docs/configuration/table.md: both operators in the field table, a Membership section (stage, textual matching, measured cast forms, refused dtypes, null rules, error codes), and an ANDed allow-list/deny-list example in one valid YAML block.
• docs/examples.md: "Identifier Allow-list and Deny-list", splitting one file into two sections with the same list under join and under antijoin. That is the shape that actually partitions the file; two different lists overlap, and a row in both is emitted twice.
• CHANGELOG.md: entry under Unreleased.

Caveats

• No zero-match warning. A list that matches nothing emits an empty section silently; catching it costs an extra pass over the frame on every build, which is a bad trade for a config typo.
• Membership is pre-resolution. The list must be spelled exactly as the source column is written, so a CURIE list needs a CURIE column. Documented in both pages.
• Deferred: file- or URL-backed identifier lists. A 40k-entry inline YAML list is the only shape today, and the op boundary (keys: list[str]) is where a loader would attach.

Testing

• uv run pytest -q --no-cov -p no:cacheprovider -> 1332 passed, 1 skipped, 2443 warnings in 21.69s (the skip is the env-gated corpus sweep, TABLASSERT_CONFIG_CORPUS).
• uv run pyright -> 0 errors, 0 warnings, 0 informations.
• uv run ruff check src tests -> All checks passed!; uv run ruff format src tests docs -> clean.
• uv run pre-commit run --all-files -> ruff check, ruff format, markdownlint-cli2, cargo-fmt, uv-lock-check, html-validate all Passed.
• uv run mkdocs build --strict -> exit 0; npx -y markdownlint-cli2@0.23.3 -> Summary: 0 issues in 0 files.
• uv run pytest -n 0 -q --no-cov tests/test_docs_source_of_truth.py tests/test_docs_examples.py tests/test_docs_style.py -> 216 passed, 1 warning.
• TABLASSERT_BENCH=1 uv run pytest -s -n 0 -q --no-cov tests/bench_reindex_membership.py -> 3 passed in 3.88s, numbers above.
• Rust: no file under rust/ changed on this branch, so cargo fmt --check, cargo clippy --all-targets -- -D warnings, and cargo test are unaffected (they were green at the merge commit).

Questions for the reviewer

• Null semantics as shipped. antijoin keeps null cells because that is anti-join semantics and it is what a deny-list section wants (blank rows stay and surface as incomplete-edges QC findings instead of vanishing). If you would rather both operators drop nulls, that is a one-line change plus two test flips, but it makes antijoin silently lose rows.
• Single PR, not a stack. The branch carries a merge of origin/main, so layering it into stacked PRs would mean rewriting that history; this is one review unit for a squash merge, matching #232 through #234 on the regex feature. Say the word if you want it decomposed.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 0f279768-ac6e-4b8f-aebc-1976807b1eff

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}