Skip to content

feat(dashboard): analytics rollups, delegation history, and trace_mismatch cue - #18

Merged
datj9 merged 3 commits into
mainfrom
feat/dashboard-analytics
Jul 26, 2026
Merged

datj9 merged 3 commits into
mainfrom
feat/dashboard-analytics

Conversation

@datj9

@datj9 datj9 commented Jul 26, 2026

Copy link
Copy Markdown
Owner

Summary

Adds the analytics surface over the telemetry #17 started persisting: per-advisor and
per-model rollups, a searchable delegation history, and a visual cue for a diagnostic that
was previously computed and then thrown away.

The headline table is check-pass rate by resolved model. That is deliberate. No
published figure survives scrutiny as evidence that routing coding subtasks to cheaper
models preserves quality — the well-known numbers are measured on chat and classification
benchmarks, not coding or agentic tool-use. So rather than assert a savings claim, this
ships the instrument that lets a user measure it on their own workload.

What's new

analytics.pybuild_analytics(jobs) computes rollups only from persisted Job
records
, never by re-parsing raw advisor logs, so the numbers can't drift from the durable
path. Per advisor and per resolved model: job and terminal counts, state distribution,
verdict distribution, check-pass rate, escalation rate, cost, tokens, duration. Computed
server-side (like graph.py) so the non-additive token rule is unit-tested in one authority
rather than reimplemented as truth in JS.

Dashboard — a third view alongside list and graph: check-pass-rate-by-model, a
by-advisor table, and a filterable history (text search + advisor + verdict) with
keyboard-accessible drill-down into the list detail.

trace_mismatch now has a visual cue. graph.py has been emitting this diagnostic
(parent exists but belongs to a different trace) while dashboard.py added only
missing_parent ids to orphanIds — so a trace-mismatched child drew as an ordinary edge
and lineage inconsistency was invisible to the operator. It now renders as a violet dotted
edge and ring, distinguished from missing_parent's red dashed treatment by both hue and
dash rhythm
, so the two aren't separable by colour alone.

Unmeasured is not zero

Only three of five advisors emit telemetry: claude (cost + tokens + duration), codex (tokens
only), commandcode (tokens + duration). opencode and gemini emit nothing.

  • Rollup metrics are None when nothing was measured and render as a muted "not
    measured"
    — never $0.00, never 0.
  • Averages exclude unmeasured jobs from the denominator. A mean cost diluted by jobs
    that never reported cost would look authoritative and be wrong.
  • A genuinely measured zero still renders as 0, and there's a test pinning that
    distinction (test_measured_zero_cost_stays_zero_not_none).
  • The UI states which advisors report what, and labels cost as a vendor-side estimate.

Tokens remain non-additive: totals sum only the primary input/output buckets; keys
naming cache or reasoning are subsets and are excluded, so cache reads aren't
double-counted (they also bill at a fraction of normal). The 6-line JS mirror of this rule
for the per-job history cell is commented as mirroring the Python authority.

Security

The rollups render advisor names, model ids, and check commands — all disk-sourced strings.
They reach the DOM only through textContent sinks (createElement + textCell), never
innerHTML with interpolated job data. Coverage extended accordingly:
test_page_escapes_job_fields_before_dom_insertion recognises the new sink,
test_textcell_sink_uses_textcontent asserts the helper writes textContent, and
/api/analytics is now included in test_prompt_never_exposed.

Accessibility

New --graph-mismatch token, defined in both theme blocks and read by the canvas via
getComputedStyle so it repaints on theme change. All ratios computed, not eyeballed:

Token Dark #bc8cff Light #8250df
on --graph-bg 8.15 4.74
on --surface 7.51 5.05
on --surface-raised 6.86 4.74

Clears the 3:1 graphical floor and the 4.5:1 text floor in both themes. No other new colour
pair — analytics reuses existing tokens. Horizontal overflow measured at 360 / 390 / 1440px:
scrollWidth - clientWidth == 0 at all three.

Test plan

  • python3 -m pytest -q406 passed (377 baseline, +29: 18 in tests/test_analytics.py,
    11 in tests/test_dashboard.py).
  • python3 -m ruff check . and ruff format --check . → both green, no unrelated files
    touched.
  • Rendered and inspected in headless Chrome at 1440 and 390px in both schemes, driving the
    real build_analytics / list_entry / build_graph output over a seeded 11-job set —
    including a job with full telemetry, one with none, a failed check, an unverified job, a
    legacy schema_version=1 record, a trace_mismatch pair and a missing_parent orphan —
    so the actual rendering code ran rather than a mock.

Known gaps

  • Escalation rate is an inferred metric. S5 (escalation ladder) hasn't landed, so there
    is no explicit escalation marker in persisted records. It is derived from lineage: a
    failed delegation counts as escalated if it has a same-trace child. This reads 0% until
    re-dispatch exists. S5 must either record retries as same-trace children or update
    analytics.py with a different definition — otherwise the column silently reports 0%
    while escalations are happening.
  • Live theme-toggle repaint of the analytics tables was not verified against a real OS
    toggle (scheme was forced via injected :root); the tables are pure CSS tokens and follow
    the media query.
  • opencode telemetry remains unmeasured upstream and renders as "not measured" rather than
    being guessed.

datj9 added 3 commits July 26, 2026 15:40
Compute cost, tokens, duration, verdict distribution, check-pass rate, and
escalation rate from persisted Job records only. Honours the measurement
contract: tokens are summed non-additively (cache/reasoning subsets excluded,
D2); unmeasured metrics stay None and are excluded from denominators, never
conflated with a measured zero (D3/D7); legacy schema 1/2 records count by
state/verdict without contributing to any measured total. Grouped per advisor
and per resolved model, with codex's absent model id landing in an explicit
not-reported bucket.
Add an Analytics view to the dashboard serving per-advisor and per-model
rollups from a new /api/analytics endpoint (analytics.build_analytics over
persisted records). Headline is check-pass rate by resolved model — the in-house
experiment for which task classes are safe to delegate cheaply — alongside cost,
tokens, duration, and escalation. Unmeasured metrics render an explicit
'not measured', never a fabricated zero, and the caveat text says averages
exclude unmeasured jobs. A searchable/filterable history table lists every past
delegation with a keyboard-accessible drill-down.

All job-derived strings (advisor, model, ids) reach the DOM only through
textContent sinks (textCell/createElement) — never innerHTML interpolation —
extending the existing XSS guard tests to the new surface.

Also render the trace_mismatch diagnostic distinctly: a violet dotted edge and
ring (--graph-mismatch token, defined in both themes; 8.15:1 dark / 4.74:1 light
on the graph background), told apart from the red dashed missing_parent orphan
cue by hue and dash rhythm. Previously trace_mismatch was computed but never
surfaced, so an inconsistent-lineage child drew as a normal edge.
The analytics rollup endpoint serves only aggregates, but add it to the
prompt-leak regression so a future field addition can't silently expose prompt
or command data through it.
@datj9
datj9 merged commit 5eb8959 into main Jul 26, 2026
6 checks passed
@datj9 datj9 self-assigned this Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants