feat(dashboard): analytics rollups, delegation history, and trace_mismatch cue - #18
Merged
Merged
Conversation
Compute cost, tokens, duration, verdict distribution, check-pass rate, and escalation rate from persisted Job records only. Honours the measurement contract: tokens are summed non-additively (cache/reasoning subsets excluded, D2); unmeasured metrics stay None and are excluded from denominators, never conflated with a measured zero (D3/D7); legacy schema 1/2 records count by state/verdict without contributing to any measured total. Grouped per advisor and per resolved model, with codex's absent model id landing in an explicit not-reported bucket.
Add an Analytics view to the dashboard serving per-advisor and per-model rollups from a new /api/analytics endpoint (analytics.build_analytics over persisted records). Headline is check-pass rate by resolved model — the in-house experiment for which task classes are safe to delegate cheaply — alongside cost, tokens, duration, and escalation. Unmeasured metrics render an explicit 'not measured', never a fabricated zero, and the caveat text says averages exclude unmeasured jobs. A searchable/filterable history table lists every past delegation with a keyboard-accessible drill-down. All job-derived strings (advisor, model, ids) reach the DOM only through textContent sinks (textCell/createElement) — never innerHTML interpolation — extending the existing XSS guard tests to the new surface. Also render the trace_mismatch diagnostic distinctly: a violet dotted edge and ring (--graph-mismatch token, defined in both themes; 8.15:1 dark / 4.74:1 light on the graph background), told apart from the red dashed missing_parent orphan cue by hue and dash rhythm. Previously trace_mismatch was computed but never surfaced, so an inconsistent-lineage child drew as a normal edge.
The analytics rollup endpoint serves only aggregates, but add it to the prompt-leak regression so a future field addition can't silently expose prompt or command data through it.
datj9-reader
approved these changes
Jul 26, 2026
This was referenced Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the analytics surface over the telemetry #17 started persisting: per-advisor and
per-model rollups, a searchable delegation history, and a visual cue for a diagnostic that
was previously computed and then thrown away.
The headline table is check-pass rate by resolved model. That is deliberate. No
published figure survives scrutiny as evidence that routing coding subtasks to cheaper
models preserves quality — the well-known numbers are measured on chat and classification
benchmarks, not coding or agentic tool-use. So rather than assert a savings claim, this
ships the instrument that lets a user measure it on their own workload.
What's new
analytics.py—build_analytics(jobs)computes rollups only from persistedJobrecords, never by re-parsing raw advisor logs, so the numbers can't drift from the durable
path. Per advisor and per resolved model: job and terminal counts, state distribution,
verdict distribution, check-pass rate, escalation rate, cost, tokens, duration. Computed
server-side (like
graph.py) so the non-additive token rule is unit-tested in one authorityrather than reimplemented as truth in JS.
Dashboard — a third view alongside list and graph: check-pass-rate-by-model, a
by-advisor table, and a filterable history (text search + advisor + verdict) with
keyboard-accessible drill-down into the list detail.
trace_mismatchnow has a visual cue.graph.pyhas been emitting this diagnostic(parent exists but belongs to a different trace) while
dashboard.pyadded onlymissing_parentids toorphanIds— so a trace-mismatched child drew as an ordinary edgeand lineage inconsistency was invisible to the operator. It now renders as a violet dotted
edge and ring, distinguished from
missing_parent's red dashed treatment by both hue anddash rhythm, so the two aren't separable by colour alone.
Unmeasured is not zero
Only three of five advisors emit telemetry: claude (cost + tokens + duration), codex (tokens
only), commandcode (tokens + duration).
opencodeandgeminiemit nothing.Nonewhen nothing was measured and render as a muted "notmeasured" — never
$0.00, never0.that never reported cost would look authoritative and be wrong.
0, and there's a test pinning thatdistinction (
test_measured_zero_cost_stays_zero_not_none).Tokens remain non-additive: totals sum only the primary input/output buckets; keys
naming
cacheorreasoningare subsets and are excluded, so cache reads aren'tdouble-counted (they also bill at a fraction of normal). The 6-line JS mirror of this rule
for the per-job history cell is commented as mirroring the Python authority.
Security
The rollups render advisor names, model ids, and check commands — all disk-sourced strings.
They reach the DOM only through
textContentsinks (createElement+textCell), neverinnerHTMLwith interpolated job data. Coverage extended accordingly:test_page_escapes_job_fields_before_dom_insertionrecognises the new sink,test_textcell_sink_uses_textcontentasserts the helper writestextContent, and/api/analyticsis now included intest_prompt_never_exposed.Accessibility
New
--graph-mismatchtoken, defined in both theme blocks and read by the canvas viagetComputedStyleso it repaints on theme change. All ratios computed, not eyeballed:#bc8cff#8250df--graph-bg--surface--surface-raisedClears the 3:1 graphical floor and the 4.5:1 text floor in both themes. No other new colour
pair — analytics reuses existing tokens. Horizontal overflow measured at 360 / 390 / 1440px:
scrollWidth - clientWidth == 0at all three.Test plan
python3 -m pytest -q→ 406 passed (377 baseline, +29: 18 intests/test_analytics.py,11 in
tests/test_dashboard.py).python3 -m ruff check .andruff format --check .→ both green, no unrelated filestouched.
real
build_analytics/list_entry/build_graphoutput over a seeded 11-job set —including a job with full telemetry, one with none, a failed check, an unverified job, a
legacy
schema_version=1record, atrace_mismatchpair and amissing_parentorphan —so the actual rendering code ran rather than a mock.
Known gaps
is no explicit escalation marker in persisted records. It is derived from lineage: a
failed delegation counts as escalated if it has a same-trace child. This reads 0% until
re-dispatch exists. S5 must either record retries as same-trace children or update
analytics.pywith a different definition — otherwise the column silently reports 0%while escalations are happening.
toggle (scheme was forced via injected
:root); the tables are pure CSS tokens and followthe media query.
opencodetelemetry remains unmeasured upstream and renders as "not measured" rather thanbeing guessed.