Skip to content

Flag learnings that name code that's gone - #731

Merged
shhdwi merged 5 commits into
mainfrom
claude/flag-stale-learnings-99579b
Nov 8, 2026
Merged

shhdwi merged 5 commits into
mainfrom
claude/flag-stale-learnings-99579b

Conversation

@anirudhkumar-nanonets

@anirudhkumar-nanonets anirudhkumar-nanonets commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

The issue: committed learnings name files and functions that were deleted since, and Claude was told to follow them as written.
Why: trail showed every learning as saved; nothing compared the code a learning names with the code that's there now.
The fix: each learning about to be shown has its touches: checked against the code in the checkout (no LLM call); one that names gone code is marked "may be stale", with what's gone and trail note --replaces <id>. The same list goes to Trail Cloud at sync.

Moves: learnings reused, measured as query events whose notes is 1 or 2 (a trail ask that showed learnings), PostHog via Metabase db 11 (posthog_event_full_view): 136,557 from 1,074 people in the 7 days to 2026-10-29 12:15 IST, out of 193,148 asks in repos that keep learnings. This PR adds stale (0/1/2) next to notes, so from the next release the same event counts how many shown learnings were stale. (learning_reused isn't used: it left the code in #658.)

What Claude sees

from <author>'s learning, a teammate's · Sep 2 · Retries to the card gateway go through withJitter, never a fixed delay
  may be stale  src/retry.ts#withJitter is gone from the code. Check it against the code before you follow it; if it no longer holds, save what's true now with trail note --replaces l-5e1f20aa
  decided     Every retry to the card gateway waits `withJitter(attempt)` (src/retry.ts): exponential…

Shown in: trail ask, MCP trail_find_code, the prompt hook, the file-touch hook, and the standing learnings at session start (as an indented may be stale: line).

The rule (one place; Trail Cloud shows what the CLI sends)

Only a learning's touches: is judged, entry by entry:

  • path is gone when the checkout has no file or folder there.
  • path#Symbol is gone when its path is, or when that file's text no longer has the name as a whole word (last part of Type.method, ~N stripped).
  • One gone entry marks the learning. A learning on a teammate's unmerged branch is never judged (its code may only be on that branch).

Trail Cloud: it has no copy of the code, so PUT /api/cli/learnings now carries missing on each HEAD learning: the gone touches verbatim, [] when all are there, left out for branch copies and skills. The sync fingerprint includes missing, so a deleted file re-sends unchanged learnings (every repo re-syncs once after upgrade, since the fingerprint changed). Today's server ignores the field (Go encoding/json, no DisallowUnknownFields), so this is safe before the trail-cloud side ships. The rule was agreed with the trail-cloud "Stale-learning cleanup" session.

What I chose, and why

  • touches: only, not names in the text. Checking backticked names in the body against code-map symbols flagged 10 of 38 real learnings (Trail-Code + assign), and 9 of those names were still in the code: properties (costUsd), module variables (livePair), Node built-ins (execFileSync). touches: only ever holds what existed when the learning was saved, so a miss there is real: on this repo's main checkout, 3 of 29 learnings are flagged, all for the deleted src/ask/jev-ask.ts.
  • The file's text, not the code map. The task said "check against the code map". I built that first, then switched: loading the map costs a hook about 0.4 s here, a fresh clone has none, and Trail Cloud can't run it. Reading the one touched file flags the same 3 learnings in about 2 ms, and both sides now state one rule.
  • Telemetry on query, not learning_reused. stale sits beside notes; allowlisted in src/telemetry/contract.ts and TELEMETRY.md.

trail check

Exits 1 with 1 conflict on src/notes/notes.ts:419, a word match ("brings back replace file name"): the new line passes the learning's id to --replaces, which is what that learning decided.

The learning this PR first added describes the code-map rule; the second one replaces it (--replaces l-d70a9e57), since learnings aren't edited.

Tests

  • New test/trail-stale-learnings.test.ts (8 tests): file and folder present vs gone, symbol as a whole word vs gone, unmerged-branch learning, the ask line, the session-start line, the sync payload's missing and the re-sync when it changes, the telemetry key.
  • Sync tests (trail-upload-safety, trail-cloud, workspace-sync-skills) pass.
  • npm test (scratch HOME): 4,143 tests, 4,131 pass, 3 fail. All 3 also fail on origin/main (19865ff) on this machine: two graft-to-trail moves and the "machine without trail" install line, which depend on whether trail is installed here.

Follow-ups

  • trail note records only files in touches:, so a learning that names a function only in its text is flagged when the file goes, not when the function does. Recording file#Symbol for backticked names the graph knows (as codeAnchors already does for init) closes that.
  • Session start could say "N of your learnings may be stale" on the count line.
  • Should a stale learning still be able to fail trail check?

🤖 Generated with Claude Code

A learning whose touches: lists a file the checkout no longer has, or a
file#Symbol no code-map symbol is named, is shown as "may be stale" in
trail ask, MCP find_code, the prompt and file-touch hooks, and the
session-start standing list. query gets a stale count next to notes.

trail check flags notes.ts:419 twice; both are word matches: the line
passes the learning's id to --replaces (what the replaces learning
decided), and the Stop-hook check-name learning is unrelated.
@trailhq-graft

trailhq-graft Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

🌱 graft blast radius

5 areas changed → 10 areas can be affected. 35 dependent symbols, depth 2.
Tests: no test reaches Workspace Sync; 2 areas updated their tests.
Tag: @Frankie-Xu — CLI · Rushabh Nagda — Workspace Sync, Cloud Collaboration · @Bortlesboat — 3 of 15 areas

flowchart TB
  A0(("Session State Management<br/>10 symbols"))
  A1(("Cloud Collaboration<br/>7 symbols"))
  A2(("Telemetry Tracking<br/>3 symbols"))
  A3(("Initialization Flow<br/>3 symbols"))
  A4(("Notes Storage<br/>3 symbols"))
  AX(("5 smaller areas<br/>9 symbols"))
  classDef reached fill:#D9EDF3,stroke:#3AA7C9,stroke-width:1.5px,color:#0E313C;
  class A0,A1,A2,A3,A4 reached;
  classDef tail fill:#EEF2F3,stroke:#9AA4A9,stroke-width:1px,color:#3A4247;
  class AX tail;
Loading
Can be affected Symbols Nearest hop Reached from
Session State Management 10 src/claude/session-metrics.ts:L1-L230 session-metrics.ts — imports, depth 1 Telemetry Contract
Cloud Collaboration 7 src/cloud/nudge.ts:L1-L508 nudge.ts — imports, depth 1 Workspace Sync, Telemetry Contract
Telemetry Tracking 3 src/telemetry/index.ts:L1-L18 index.ts — imports, depth 1 Telemetry Contract
Initialization Flow 3 src/init/keep.ts:L1-L387 keep.ts — imports, depth 1 Telemetry Contract
Notes Storage 3 src/notes/memory-notes.ts:L1-L285 memory-notes.ts — imports, depth 1 Telemetry Contract
Ask Interface 3 src/ask/ask.ts:L1492-L1544 renderAsk — calls, depth 1 Notes Retrieval
4 smaller areas 6 MCP Integration, Engine Core, CLI Skills, … see below
Who knows this code — 11 people across 15 areas
Area Who knows it
Workspace Sync · changed Rushabh Nagda — 2 commits, last 18d ago · @Faeeth — 1 commit, last today
Prompt Formatting · changed Ryan Duguid — 1 commit, last yesterday
CLI · changed @Frankie-Xu — 4 commits, last 17d ago
Session State Management · affected @Bortlesboat — 1 commit, last yesterday · @Rakly3 — 1 commit, last 17d ago
Cloud Collaboration · affected Rushabh Nagda — 1 commit, last 19d ago
Telemetry Tracking · affected @afeddersen — 1 commit, last 2mo ago
Notes Storage · affected @Faeeth — 1 commit, last today
Ask Interface · affected Buseong Kim — 3 commits, last 2mo ago · @VivienP — 1 commit, last 3mo ago
…7 further areas

Ownership is git history over each area's own files, weighted towards recent work (120-day half-life). Merge commits and bots are dropped, and you are dropped from your own PR. A name with no @ has no GitHub handle in its commit email — tag them by hand, or add a .mailmap entry. A suggestion from history, not a CODEOWNERS rule.

All 35 dependent symbols, grouped by area

Session State Management — 10 symbols in 10 files

  • src/claude/session-metrics.ts:L1-L230 — session-metrics.ts (imports, depth 1)
    67: * "graft" (or a repo working on graft itself) must not count as a graft read —
  • src/claude/skills-wiring.ts:L1-L116 — skills-wiring.ts (imports, depth 1)
    16: import { hookMsBucket } from '../telemetry/contract.js';
  • src/claude/state.ts:L1-L165 — state.ts (imports, depth 1)
    33: /** The repo skills this session used, by name, to when it last went to the workspace as a run (skill-use.ts). */
  • src/claude/hook-io.ts:L1-L140 — hook-io.ts (imports, depth 2)
    17: /** The env a hook's graft child runs with. In a repo on trail with no map yet, the
  • src/claude/prompt-skills.ts:L1-L327 — prompt-skills.ts (imports, depth 2)
    4: * skill a prompt brings back. `handlePrompt` calls it right after the
  • src/claude/skill-use.ts:L1-L80 — skill-use.ts (imports, depth 2)
    2: * A coding agent using one of the repo's skills, as a run on the workspace.
  • src/claude/stats.ts:L1-L27 — stats.ts (imports, depth 2)
  • src/claude/statusline.ts:L1-L40 — statusline.ts (imports, depth 2)
    3: import { renderStatusline, renderSubagent } from './format.js';
  • src/claude/stop.ts:L1-L216 — stop.ts (imports, depth 2)
    11: * to save a learning (note-signals.ts). An offer or notice that wins leaves
  • src/claude/sync-run.ts:L1-L77 — sync-run.ts (imports, depth 2)
    65: // An optional build timeout in ms: a repo's first build, started by a query

Cloud Collaboration — 7 symbols in 4 files

  • src/cloud/nudge.ts:L1-L508 — nudge.ts (imports, depth 1)
    5: * sharing, `trail ask`, `trail note` and `trail check` hand the agent a link
  • src/cloud/lookup-activity.ts:L1-L60 — lookup-activity.ts (imports, depth 2)
    8: import { noteAuthor } from "../notes/git-facts.js";
  • src/cloud/other-repos.ts:L102-L135 — setUpPicked (calls, depth 2)
  • src/cloud/pair.ts:L209-L242 — finish (calls, depth 2)
    209: finish(repo: string, found: FoundSkill[] = []): Promise<void> {
  • src/cloud/pair.ts:L137-L155 — start (calls, depth 2)
    137: static async start(repo: string, code: string, opts: { agent: string; version: string; fetchImpl?: typeof fetch }): Promise<PairRun | { error: string; status?: number }> {
  • src/cloud/pair.ts:L308-L367 — waitAndUpload (calls, depth 2)
    308: async function waitAndUpload(state: string, repo: string, opts: FinishOptions): Promise<Waited> {
  • src/cloud/pair.ts:L1-L368 — pair.ts (imports, depth 2)
    12: * review, and uploads the repo the usual way, which leaves a dropped learning

Telemetry Tracking — 3 symbols in 3 files

  • src/telemetry/index.ts:L1-L18 — index.ts (imports, depth 1)
    2: * Anonymous adoption telemetry. `TELEMETRY.md` is the public contract;
  • src/telemetry/sessions.ts:L1-L113 — sessions.ts (imports, depth 1)
    21: * must never touch the network (see the note atop upkeep.ts) and this doesn't:
  • src/telemetry/track.ts:L1-L163 — track.ts (imports, depth 1)
    42: repo?: string;

Initialization Flow — 3 symbols in 3 files

  • src/init/keep.ts:L1-L387 — keep.ts (imports, depth 1)
    9: * - A guardrail: the catalog's template (`guardrailLearning`), every repo
  • src/init/run.ts:L1-L483 — run.ts (imports, depth 1)
    3: * no-model look-back in a worker while init wires the repo, then show what
  • src/init/review-screen.ts:L1-L319 — review-screen.ts (imports, depth 2)
    44: found: '✓ looked back at {n} Claude Code sessions about this repo, typed into on {d} days ({from} to {to})',

Notes Storage — 3 symbols in 3 files

  • src/notes/memory-notes.ts:L1-L285 — memory-notes.ts (imports, depth 1)
    2: * Learnings from the notes Claude Code keeps for this repo on this machine
  • src/notes/pr-reviews.ts:L1-L577 — pr-reviews.ts (imports, depth 1)
    2: * Learnings from a repo's pull requests, for `trail init` to offer on day one
  • src/notes/claude-export.ts:L1-L128 — claude-export.ts (imports, depth 2)
    11: * screen's r adds one to the repo. Facts the repo already says, and facts

Ask Interface — 3 symbols in 1 file

  • src/ask/ask.ts:L1492-L1544 — renderAsk (calls, depth 1)
    1494: // The note prints as its own prominent line(s) right under the header —
  • src/ask/ask.ts:L1561-L1564 — askSavedTokens (calls, depth 2)
  • src/ask/ask.ts:L1552-L1557 — formatAskWithSaved (calls, depth 2)

MCP Integration — 3 symbols in 2 files

  • src/mcp/server.ts:L1-L257 — server.ts (imports, depth 2)
    17: import { KeptListing, within } from './kept-listing.js';
  • src/mcp/tools.ts:L265-L351 — callSingleTool (calls, depth 2)
    289: return { text: formatSkeleton(r), isError: !r.entries.length && !!r.note };
  • src/mcp/tools.ts:L1-L352 — tools.ts (imports, depth 2)
    50: 'Query the repo context graph in plain words. Returns ranked nodes with exact file:line spans and the relevant source inlined — usually the full answer, no file reads needed.',

Engine Core — 1 symbol in 1 file

  • src/engine.ts:L115-L125 — ask (calls, depth 1)
    124: return attachNotes(r, dir);

CLI Skills — 1 symbol in 1 file

  • src/skills/cli-skills.ts:L1-L376 — cli-skills.ts (imports, depth 2)
    118: /** A personal skill by name (any repo), or a name the person turned off. */

Learning Sessions — 1 symbol in 1 file

  • src/learn/sessions.ts:L1-L310 — sessions.ts (imports, depth 1)
    2: * Learnings from a repo's past Claude Code sessions, for `trail init`: what
Test signal per changed area — 2 ✓ · 1 ✗ · 2 –

Reached = a node under a test path has a resolved edge into the changed symbol. It undercounts anything called indirectly — through a CLI, a spawned process or a dynamic import — so read a low ratio as “look here”, never as a coverage gate.

  • – Telemetry Contract — 1 test file changed here: test/trail-stale-learnings.test.ts
  • ✓ Notes Retrieval — 4 of 9 reached · 1 test file changed here: test/trail-stale-learnings.test.ts
    • not reached: staleLine, hasWord, inCheckout, symbolName, within
  • ✗ Workspace Sync — 0 of 2 reached · no test file reaches it
    • not reached: workspaceLearnings, fingerprint
  • ✓ Prompt Formatting — 3 of 5 reached · 1 test file changed here: test/trail-stale-learnings.test.ts
    • not reached: learningsOnTouch, learningsAtStart
  • – CLI — no function, method or class changed here
68 test suites also reference this code

71 symbols, kept out of the diagram and the table so they cannot crowd out the areas a reviewer has to look at.

  • test/claude-copy-dashes.test.ts
  • test/claude-export.test.ts
  • test/claude-format.test.ts
  • test/claude-hook-timeouts.test.ts
  • test/claude-hooks-1b.test.ts
  • test/claude-hooks-perf.test.ts
  • test/claude-hooks-skills.test.ts
  • test/claude-hooks.test.ts
  • test/claude-host-standdown.test.ts
  • test/claude-post-bash-walls.test.ts
  • test/claude-prompt-learnings.test.ts
  • test/claude-prompt-private.test.ts
  • test/claude-session-start-learnings.test.ts
  • test/claude-skills-journey-fixtures.ts
  • test/claude-start-skills-1b.test.ts
  • test/claude-state.test.ts
  • test/claude-statusline.test.ts
  • test/claude-stop-trace.test.ts
  • test/claude-stop.test.ts
  • test/claude-tally.test.ts
  • …48 more

⚠️ 3 changed files not in the graph (.trail/learnings/2026-10-29-the-stale-learning-check-judges-touches-anirudhkumar.md, .trail/learnings/2026-10-29-the-stale-learning-check-reads-the-anirudhkumar.md, TELEMETRY.md) — no parser claims the extension, or the index predates the file.

graft blast · origin/main...HEAD · depth 2 · 14 changed files

Open the interactive graph → — click an area to see its dependent symbols at file:line.

github-actions Bot added a commit that referenced this pull request Oct 29, 2026
Trail Cloud has no code, so the CLI does the check and sends the gone
touches per learning (`missing`, [] when all are there); the sync goes
again when that list changes. The symbol check reads the touched file's
text instead of the code map, the rule both sides now state the same
way, and drops the 0.4s map load from hooks.
@shhdwi

shhdwi commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

Reproduced on main: after renaming withJitter away, trail ask still showed the learning as if it held. On this branch it says "may be stale", names what's gone and gives the --replaces id. Tests pass.

Main has moved, though: #747 changed learningsOnTouch in src/claude/hooks.ts, so this conflicts now. Please merge main and keep both: the addHookSavings(s, text) call from #747 and your TOUCH_NOTES_CAP, dir arguments.

Heads-up for whichever of this and #729 lands second: both add a different 5th parameter to notesOnTouch in src/claude/format.ts.

Shrish Dwivedi and others added 3 commits November 8, 2026 16:19
Keeps both sides of learningsOnTouch: addHookSavings from #747 and the
TOUCH_NOTES_CAP, dir arguments that mark stale learnings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
touches: comes from teammates' learning files, and what the stale check
finds goes up at sync as missing. A ../ path, an absolute one or a symlink
out of the repo could have probed any file on the machine (whether it
exists, whether it holds a word). Resolve each path, follow symlinks, and
skip it unless it stays inside the checkout. Also skip reading files over
2 MB for a symbol, and drop a private repo's name from the learnings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
github-actions Bot added a commit that referenced this pull request Nov 8, 2026
@shhdwi
shhdwi merged commit 6e64f33 into main Nov 8, 2026
8 checks passed
@shhdwi

shhdwi commented Nov 8, 2026

Copy link
Copy Markdown
Collaborator

Merged with these commits on top:

  • Merged main. In learningsOnTouch I kept both sides: notesOnTouch(notes, touch, s, TOUCH_NOTES_CAP, dir) and the addHookSavings(s, text) call from Keep the tokens-saved counter moving in long sessions #747.
  • Security fix: touches: comes from teammates' learning files, and the stale check reads the file each one names, then sends what's missing up at sync. A ../ path, an absolute path or a symlink out of the repo could have checked any file on the machine (does it exist, does it contain a word). Each path is now resolved, symlinks included, and skipped unless it stays inside the checkout. Files over 2 MB aren't read for a symbol either, since a hook reads them whole. A new test in trail-stale-learnings.test.ts covers ../, absolute and symlinked paths.
  • Took a private repo's name out of the two learnings.

Repro in a scratch repo with a scratch HOME: a learning that names src/gone.ts plus two ../outside/... paths. trail ask flags only src/gone.ts as maybe stale. Ubuntu CI passed.

github-actions Bot added a commit that referenced this pull request Nov 8, 2026
shhdwi pushed a commit that referenced this pull request Nov 8, 2026
notesOnTouch takes dir (from #731) before view. learningsOnTouch keeps
#731's TOUCH_NOTES_CAP, dir arguments; the savings now come from
notesOnTouch itself (previous commit), so addHookSavings goes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shhdwi pushed a commit that referenced this pull request Nov 8, 2026
…ale mark on one line

notesOnTouch now takes dir before view, so the tests pass undefined for
it. One-line mode dropped #731's "may be stale" warning entirely; it now
says "may be stale" beside that learning's path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shhdwi pushed a commit that referenced this pull request Nov 8, 2026
* Add trail summary one-line for the hook blocks

* Count hook learnings' savings from the learnings, in either view

Since #747 the tokens-saved counter read the savings line back out of the
hook's text, and one-line mode has no such line, so with one-line on the
counter stopped moving for learnings a hook brought. notesForPrompt and
notesOnTouch now add learningSavedTokens(hits) to the session themselves,
in both views, and the hooks no longer parse their own text. A learning
title that reads like a savings line can't move the counter either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix summary-view.test.ts for notesOnTouch's dir argument; keep the stale mark on one line

notesOnTouch now takes dir before view, so the tests pass undefined for
it. One-line mode dropped #731's "may be stale" warning entirely; it now
says "may be stale" beside that learning's path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Shrish Dwivedi <shrish@nanonets.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants