fix(executor): semantic_deduplicate via MinHash + LSH (was O(n²), hung run 8afd8d39)

[DHYEY166]

Root cause

semantic_deduplicate (lambdas/executor/handler.py) compared each row's MinHash signature with every signature kept so far, element by element in pure Python: O(n² · num_perm). Run 8afd8d39 (56,745-row tweet CSV, num_perm 120, threshold 0.9) would need hours. The executor Lambda is killed at 15 minutes, so the run stayed on Running, timed out, was redelivered by SQS, and finally failed. Nothing in the rule watched the clock.

There was a second bug. Signatures used Python's built-in hash(), which PYTHONHASHSEED randomizes per process, so the same file could dedupe differently on every invocation.

Fix

• Deterministic MinHash, vectorized in numpy. Tokens are hashed the same way as before (lowercased whitespace-split set) using blake2b (32-bit). Each of the num_perm permutations is a fixed universal hash ((a·x + b) mod 2⁶¹−1) mod 2³², with a/b derived from blake2b. All values stay below 2³², so a·x + b is exact in uint64. Per-row minima come from np.minimum.reduceat, chunked to about 16 MB of intermediates. There is no new dependency: numpy already comes with the AWSSDKPandas layer.
• LSH banding. r is the largest band width whose b = num_perm // r bands make a pair at the threshold a candidate with probability ≥ 99%. Examples: 0.9/120 → 12 bands × 10, 0.8/64 → 12 × 5, 0.8/128 → 21 × 6. Rows are walked in order. For each row, only the earlier kept rows that share a bucket are candidates, and each candidate is verified with the full-signature estimate matches/num_perm ≥ threshold. That is the same decision rule as before, so the first occurrence is still kept. A test checks the output is identical to brute-force pairwise comparison on the same signatures.
• Guards (the rule is skipped, never hangs):
  • handler() records context.get_remaining_time_in_millis(). If hashing or comparison would run into the last min(EXECUTOR_RESERVE_S=120s, 25%) of the invocation, the rule raises RuleSkipped. The frame is restored and the reason shows as "Not applied", for example semantic_deduplicate skipped: comparing 56,745 rows would exceed the Lambda time limit …. The rest of the run still completes.
  • There is a row cap, SEMANTIC_DEDUP_MAX_ROWS (default 500,000).
  • Without a Lambda context (local runs or tests), the rule has a 600 s budget.
  • An invalid threshold (outside (0, 1]) or num_perm (outside 1..512) is skipped with a reason.

Complexity

	Before	After
Signatures	O(n · tokens · num_perm), Python loops	Same order, numpy-vectorized
Comparison	O(n · kept · num_perm), Python	O(n · bands) dict lookups + O(candidates · num_perm) numpy checks
4,000 short texts (0.9/120)	33 s	0.5 s
50,000 short texts (0.9/120)	~1.5 h (extrapolated from the 1k/2k/4k quadratic fit)	2.3 s (Python 3.13, pandas 3) / 4.1 s (3.12, pandas 2)
56,745 rows	hours, so a Lambda timeout	~3 s

Behaviour change

• The decision rule is unchanged, but the hash functions are new, so rows near the threshold can come out differently than in an old run (before, that varied per process anyway).
• On 3,000 synthetic rows with many pairs near the threshold, results match an exact-Jaccard dedup apart from MinHash estimation noise right at the threshold. That noise is inherent to MinHash and was there before too. LSH itself added no misses there: the output is identical to the brute-force MinHash comparison.
• Invalid threshold/num_perm values are now skipped instead of silently keeping or dropping everything.

Tests

New file lambdas/tests/test_semantic_dedup.py (23 tests). There were no tests for this rule before.

• Behaviour: near-duplicates dropped and first occurrence kept, below-threshold rows kept, default text column, no text column, invalid params.
• LSH: output equals brute-force pairwise; LSH parameters reach ≥ 0.99 candidate probability at the threshold.
• Determinism: pinned golden signature digest; same digest in a subprocess with a different PYTHONHASHSEED; same rows on repeat.
• Guards: row cap, near-expired Lambda deadline (frame restored), reserve arithmetic, handler() records the deadline.
• Perf: @pytest.mark.perf, 50k short texts with a generous bound of < 60 s (takes 2–5 s).

Locally: pytest lambdas/tests gives 86 passed and 5 skipped (the 5 integration tests need services) under both pandas 2.2.3/numpy 2.1.3 (Python 3.12) and pandas 3.0.6/numpy 2.5.3.

Notes

• README.md: updated the rule table, the executor env vars and the known limitation.
• The guard covers only semantic_deduplicate. Other rules and the output write have no deadline check, and there is still no upload size limit.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
clean-stack	Ready	Preview	Oct 2, 2026 3:04pm UTC
cleanstack	Ready	Preview, v0	Oct 2, 2026 3:04pm UTC