search_events: smoke test is 66% of rows and makes retrieval metrics read backwards

[mkreyman]

Found while reviewing every place the harness tells an agent to search the KB. The search_events instrumentation (shipped today) is working, but two things make it read wrong, and one of them silently corrupts the headline retrieval metrics.

1. The smoke test is 66% of all recorded searches

scripts/smoke.sh lines 447 and 455 issue two searches per run with q=elixir — one mode=keyword, one default. In the first day of data that is 135 of 204 rows, matching exactly in the DB: 67 keyword + 64 combined, in pairs, ~2/min in bursts.

Effect on how the table reads:

segment	rows	avg query_terms
infra (client_host IS NULL — smoke + UserPromptSubmit hook)	192	2.7
agent via MCP (client_host IS NOT NULL)	12	9.5

Un-segmented, 71% of queries look like single-token junk and the natural conclusion is "agents write terrible queries." Segmented, agent queries average 9.5 terms and are specific and topical. The conclusion reverses.

docs/runbooks/search-events-analysis.md already documents the client_host IS NOT NULL trap in step 2, and there is a KB article on it (80d5c9b0-7deb-4726-b337-3cb7e5efcfdb). The problem is that nothing enforces it — the default reading of the table is the wrong one.

Ask: make the segmentation the default rather than a documented caveat. Options, cheapest first:

• Tag smoke traffic explicitly (a source/client_entrypoint value of smoke, or a distinct api_key for scripts/smoke.sh) so it can be excluded by a predicate that does not rely on a NULL meaning two different things. client_host IS NULL currently conflates smoke test with legitimate hook-driven recall, which are very different populations and should not share a bucket.
• Exclude tagged infra traffic from knowledge_retrieval_metrics by default, with an opt-in to include it.

2. knowledge_retrieval_metrics is currently unreadable because of the above

Measured over the last 10 days:

day	searches	results surfaced	opened	precision	search_follow_through
08-11	227	1137	12	1.1%	4.0%
08-10	112	564	5	0.9%	3.6%
08-09	77	377	15	4.0%	15.6%
08-07	288	1550	37	2.4%	10.4%

A 1–4% precision reads as a catastrophic retrieval failure. But the smoke test searches and never opens, by construction — every one of its runs adds to the denominator and can never add to the numerator. Until infra traffic is excluded, these numbers cannot be used to judge retrieval quality, and any work prioritised off them is prioritised off an artifact.

The tool's own docstring is careful about several biases; this one is not among them and is currently the largest.

3. Smaller: memory_recall rows carrying a bare repo name

Seven rows are single-token memory_recall calls — loopctl, infra, home_care_billing, .claude, claude-config. That is the SessionStart hook scoping by project, which is legitimate, but it lands in the same query/query_terms columns as real searches and skews the query-shape analysis the column was added for. Worth either excluding scoping calls from query-shape stats or recording them with a distinct tool/mode.

Why this matters beyond tidiness

The instrumentation was added to answer "are agents searching badly?". On the first day of data the honest answer is no — the agent-authored queries are 8–15 terms and topical. If the table is read without segmentation it says the opposite, and the fix effort goes to agent prompting instead of to retrieval, which is where the evidence actually points.

Context

Raised from a claude-config-side review of all 43 harness call sites that instruct an agent to search. The claude-config half is handled separately (mkreyman/claude-config#263): the recall hook's relevance floor now logs its decision, and a 13-prompt calibration through the real hook path shows real/junk top scores have converged (REAL 0.372–0.600, JUNK 0.005–0.456) so an absolute score floor no longer separates them.

[mkreyman]

Direction change from the owner: drop the threshold, let the querying agent judge

Rather than recalibrating the client-side relevance floor, the intent is now to remove the concept and leave relevance judgement to the agent reading the pack. The measurement above is the argument for it: a scalar threshold is the machine guessing relevance from a number that no longer discriminates (REAL 0.372–0.600 vs JUNK 0.005–0.456, overlapping), and it currently fails in both directions at once — suppressing whole real packs while still leaking junk rows.

An agent reading the block knows the task, the topic, and what it is looking for. It can judge a row in context; a constant cannot.

What that needs from loopctl. If the client stops thresholding, the rows must arrive labelled well enough to be judged without opening each one. Two asks:

1. Give /api/v1/recall the provenance/confidence signal the MCP surface already has. knowledge_hybrid_search returns meta.provenance (curated vs retrieved) plus a confidence — that is a genuine no-answer-ish signal, and far more useful to a reading agent than a raw score. /api/v1/recall returns a bare 0..1 score with no calibration and no way to express "nothing here really answers this."

2. A explicit low-confidence / no-answer indication. The root problem named in the original finding is that semantic search has no no-answer mode, so a procedural turn still gets three nearest neighbours. If the server can say "these are weak neighbours, not answers," the client can render that honestly and the agent can skip them — which is the same outcome the floor was trying to achieve, but by informing the reader instead of silently deleting rows.

Why this is a better place for the logic than the client. The server knows the score distribution across the whole corpus and can express confidence relative to it; the hook only ever sees five rows from one query and has to hard-code a constant that goes stale the moment fusion, embeddings or corpus size change — which is exactly what happened.

The claude-config side (removing the floor, rendering rows with honest labels, and stopping the unopened-row counter from retiring articles that were merely surfaced on a procedural turn) is tracked separately; it is blocked on nothing, but it is materially better with (1) and (2) than without.

[mkreyman]

Verified live in prod, not waiting for tomorrow

Ran the whole chain against production rather than waiting a day to find out it didn't work.

All three classes are recording, within 30 minutes of the deploy:

entrypoint	host	tool	rows
smoke	(none)	knowledge_search	12
hook	(none)	memory_recall	1
cli	beelink	knowledge_search	2
sdk-cli	Marks-Mac-mini	knowledge_search	4

The exclusion uses it, with exact arithmetic. RetrievalMetrics.compute/3 against prod
reported searched=1450, and:

1474  all surfaced search rows today
  24  tagged smoke        <- excluded
1450  remainder            <- matches what the metric reported

So tag -> propagation into article_access_events -> exclusion is confirmed end to end.

A prediction, so nobody misreads the step change

Tagging only began at 17:41, so most of today is still untagged. Today is 41.1% smoke:
24 tagged (excluded) plus 582 untagged carrying the q=elixir signature (still counted).

Projecting a full day of tagging, with no retrieval change at all:

	today	tomorrow
denominator	1450	~868
opens	20	20 (smoke never opens, so the numerator does not move)
precision	1.38%	~2.3%

Precision should rise about 1.7x overnight and it will mean nothing about retrieval
quality. It is the artifact leaving the denominator. Do not read the jump as improvement,
and do not compare any day before 2026-08-12 17:41 with any day after it — the populations
differ.

The first genuinely comparable series starts 2026-08-13.