Keep rows with placeholder cells, report row removals, guard bad-cell filters

[DHYEY166]

Fixes the issues from a small xlsx run where every suggested rule was approved: the Brown row (. in its three numeric cells) disappeared, three numeric columns got orig_<col> copies in the deliverable, and the COLLEGEPOND REFERENCE label row was gone. Adds a deterministic guard for row-removing AI suggestions.

Root causes

1. Brown: no type cast drops rows (type_cast turns . into null and keeps the row). Brown could only be removed by a row filter on a ranking column. Two paths in the old code: a neq "." filter (the prompt told the model to "filter to exclude" sentinels), or a comparison filter (gt 0), which treated . and nulls as failing the comparison. Nothing recorded how many rows a rule removed, so the loss was silent.
2. orig_<col> columns: trim_whitespace turned numbers into strings (34 became "34"), so the later cast wrote __orig_ sidecars for every value. column_header_normalize then snake-cased __orig_X to orig_x, so split_deliverable no longer saw them as sidecars and they leaked into output.xlsx.
3. COLLEGEPOND REFERENCE: not a header-detection bug. Both the executor and the profiler read it as the first data row (tests pin this). The suggested rules with no value filters keep it, so it was removed by an approved rule that targeted rows by value. It is the same class of problem as 1, and the new per-rule rows_removed makes it visible.

Changes

Executor (lambdas/executor/handler.py)

• rows_removed per rule goes into _execution and the log.
• type_cast accepts null_values (placeholder codes become null and the row is kept).
• Comparison filters keep rows whose value is missing or unparseable.
• trim_whitespace strips only text.
• Sidecars are kept only when a value lost information. rename and column_header_normalize move the sidecar with its column.
• Bad-cell guard (bad_cell_rows()): counts the rows a filter/filter_extended/drop_nulls removed only because of one placeholder or non-number cell in an otherwise valid row (at least half of the other cells filled in). This is recorded as _execution.bad_cell_rows. In auto mode such a rule is skipped and the rows are kept. A rule a person approved still runs and is reported.

App

• src/lib/rule-guard.ts: after the AI returns rules and before they are stored, flagged rules get parameters._guard (reason plus a type_cast/null_values alternative). The rule card shows the warning, "Approve all" skips flagged rules, and auto-validate never approves one.
• Prompt: a single bullet changed. Sentinels in a mostly numeric column now map to type_cast to null, with a filter only when the whole row is invalid. This is generic, with no dataset-specific tokens.
• Run page: rows removed per rule and in the summary, plus "Removed for one bad cell" and "Kept by bad-cell guard" counts.

The guard does not flag legitimate filters: comparisons, keep-only matches (eq/in/regex), numeric values, real text values (neq "deleted", not_in ["test"]), blank removal, or mostly empty rows. PLACEHOLDER_TOKENS is shared between TS and Python and checked by a contract test.

Tests

• lambdas/tests/test_college_list.py: fixture shaped like the real file.
• lambdas/tests/test_bad_cell_guard.py and src/lib/rule-guard.test.ts: flagged and not-flagged cases.
• Route tests for suggest-transforms (guard wiring) and auto-validate (flagged rules are rejected).
• The fake model's rules are unchanged (trim/type_cast/normalize), and integration and e2e pass.

Docs: docs/transform-rules.md.

After merge: redeploy the executor Lambda (zip-swap). The profiler is unchanged.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
clean-stack	Ready	Preview	Oct 3, 2026 2:09pm UTC