Ship a Claude Code skill so the agent searches its own history - #55
Merged
Conversation
agsearch has been a thing you run beside a coding agent. As a skill it is
something the agent runs itself: you refer to an earlier conversation, and it
searches the transcripts on disk and answers from them instead of telling you
it has no record of it.
The skill is one markdown file. It ships in `skills/agsearch/SKILL.md` where
people can read it before installing, and as a literal inside the script,
because brew, uv and the curl installer each put exactly one file on your PATH
and none of them carries a data directory. tools/sync-skill.py copies one to the
other and a test fails when they drift.
What the skill teaches is query construction, because that is the one place an
agent reliably fails. Measured over 247 queries: content words alone rank the
right session first 49% of the time, the same words left inside the question
that carried them score 10% to 22%. Extending the stoplist to absorb question
phrasing was tried and rejected: it recovered a few points at best, and the
words still doing the damage ("decide", "session", "discuss") are real search
terms in other queries, so stoplisting them fits the benchmark rather than the
corpus.
Two behaviours came out of watching agents run against a real corpus rather
than from reasoning. They fall back to grepping ~/.claude/projects, where JSONL
tool calls and base64 attachments produce matches nobody said, so the skill says
to go through agsearch. And on a handoff they reach for git first, which is
correct as far as it goes, so the skill says the transcript carries the
reasoning and the discarded options while git carries what shipped.
A capped `read` given search terms was keeping the last turns and dropping the
matched ones, which answers a question nobody asked. Matched turns now outrank
the ending, and on one session that took the kept matches from 8 to 27 inside
the same budget.
`--install-skill` writes it to ~/.claude/skills/agsearch/ and leaves an edited
copy alone unless forced; the curl installer does the same, honouring
AGSEARCH_SKILL=0. CI now fails if an install does not carry the skill.
Closes #46.
The skill said the top hit is right about three times in four. That is the hand-labelled list (0.733); the 247-query set nobody tuned against reads 0.490. The instruction that matters is to check the matched line either way, and it does not need a number to land.
The skill existed twice: as skills/agsearch/SKILL.md and as a literal inside the script, with a sync tool and a drift test holding them together. That was built to work around brew, uv and curl each installing exactly one file, but a skill is a user-level config file, not a program asset, and the ecosystem already has a way to install one. Two manifests replace all of it. `/plugin marketplace add devcodes9/agsearch` and `/plugin install agsearch@agsearch` install the skill from this repo, and `plugin.json`'s version controls when users get updates, so the staleness problem the sync test was guarding against stops existing. Copying skills/agsearch/ into ~/.claude/skills/ still works for anyone who would rather not add a marketplace. Removed: the SKILL_MD literal, install_skill(), --install-skill, the install.sh block, tools/sync-skill.py, and the CI step that checked the install carried the skill. Net effect on the script is that it goes back to being a search tool that does not know what a skill is. Verified with `claude plugin validate .`, then installed locally end to end: one skill, ~106 tokens always-on. The manifests are now tested rather than the drift: the marketplace lists the plugin, its source directory exists, that directory really contains a skill, and the version is a version.
The plugin is a markdown file that installs on its own, so someone can hold the skill without ever having installed the CLI it drives. Every command then fails with command-not-found and the skill said nothing about it, which leaves the agent to retry or to go reading transcripts by hand. Names the install commands once and says to stop rather than work around it.
The plugin sets source to the repo root, so installing it also puts the agsearch script in the plugin cache, executable and working. The skill was telling the agent to give up when the PATH copy was missing, while a usable binary sat next to the skill file. Now: PATH first, then $CLAUDE_PLUGIN_ROOT/agsearch, and only then say it is missing. $CLAUDE_PLUGIN_ROOT is how other plugins reach their bundled files, so this is the existing convention rather than a new one. Checked how 123 locally installed skills handle a shelled-out dependency: none carry "if it is not installed" prose, and plugin.json has no dependency field to declare one with. Bundling and pointing at it is what the format actually supports.
Audited against superpowers:writing-skills and mattpocock's writing-for-agents, using their no-op test: delete a line, and if the agent would do the same thing, the line was only costing context. The description opened by saying what the skill does. Both guides are explicit that a description carries triggering conditions only, so it now leads with when to reach for this and names the harnesses at the end. The first paragraph then restated that description, which put the harness list in two places to drift apart. Only the sentence that changes behaviour is left: the one that answers the model's instinct to say it has no record. "Never read the transcripts directly" was filed under reading results, three sections away from the command it constrains. Moved next to it. Dropped that everything runs locally and nothing leaves the machine. That is a fact for a human choosing whether to install this, and there is no branch where an agent does anything differently for knowing it.
A description is a guess about when an agent will reach for a skill until it is measured. Twenty cases in the format the official math-olympiad plugin uses, ten that should fire and ten that should not. The negatives carry the weight. Half of them say earlier, before, previous or we discussed while meaning this session, this file, or git history, which is the way this description is most likely to over-fire.
`claude plugin eval` is early access, so nothing here can run this file yet, and its cases are written from one person's own sessions rather than for the class of task. It belongs beside the runner until the eval command is generally available and the cases are worth publishing.
The no-hits branch told the agent to stop and report the conversation absent. That is the wrong answer twice over: a lexical miss means those words are in no transcript, not that the conversation never happened, and the user usually holds the one detail that would find it. Measured on the 247-query held-out set, against the 126 queries that miss rank 1 today. A project hint recovers 31% of them and a seven-day date window 39%, but the date has to be right to within a week: a month-wide window recovers 13% and a quarter-wide one nothing. Ask for the repository. It is nearly as strong, people recall a repo far better than a week, and --project already exists. Also the rule that makes filtering safe: never conclude a conversation is absent from a filtered search. A wrong hint removes the target and turns a buried result into a confident "not found", which is worse than the miss it replaced. Skill text only. No code changes.
The skill did not fire. Asked "what did we decide about the webhook retry backoff?" with the plugin enabled and the skill loaded, the agent read MEMORY.md, asked an unrelated memory server, then grepped ~/.claude/projects by hand, and answered that it had no record. Three runs, three misses, a different improvised strategy each time. Tidying the description did not help, because the problem was never the wording of the trigger. The skill already forbids reading the transcripts directly, but that sentence lives in the body, which only loads once the skill has fired. The one instruction that could have prevented the failure was behind the door it was meant to open. So the description now names the competing behaviour. On the same prompt the skill fires, and it fires on a second prompt twice more. Found by running it. The audit that came before this read the file against two style guides and rated it healthy, which it was, and which changed nothing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
agsearch has been something you run beside a coding agent. This makes it something the
agent runs itself: you mention an earlier conversation, and it searches the transcripts
already on disk and answers from them, instead of telling you it has no record of it.
Closes #46.
What ships
skills/agsearch/SKILL.md, installed as a Claude Code plugin. It covers two workflows:continue it.
claude --resumeputs you back in the old session in its own directory;this carries that session's context forward into the one you are in now.
How it installs
Two manifests, no code.
plugin.json's version controls when users get an update, so thereis nothing to keep in sync and nothing to go stale. Anyone who would rather not add a
marketplace can copy
skills/agsearch/into~/.claude/skills/instead.The skill needs the
agsearchbinary, which is unchanged by this.Verified with
claude plugin validate .and installed end to end locally: one skill,~106 tokens always-on, ~1.1k when it fires.
What the skill teaches, and why
Query construction, because that is where an agent fails. Measured over 247 queries drawn
from the corpus rather than hand-picked:
what did we decide about ...we hit ... before, what happened therecan you find the session where ... and continue itwhere did we discuss ...So the skill's central instruction is to pass the nouns, error strings and identifiers,
not the question that carried them.
Rejected: fixing this in the ranker instead
Extending the stopword list to absorb question phrasing looked like the better fix, since
it would help every caller rather than only an agent holding the skill. Measured, it
recovers very little:
what did we decide aboutmoves 0.215 to 0.279 andwhere did we discuss0.101 to 0.117, with bare queries and the hand-labelled list unchanged. The wordsstill doing the damage are decide, discuss, session and continue, which are content
words in other people's queries. Stoplisting them fits the benchmark rather than the
corpus, so this is not shipped.
Two changes that came from watching agents rather than from reasoning
Run against a real corpus of ~740 sessions, agents did two things I had not predicted.
They grep the transcripts by hand. Those files are JSONL with tool calls, diffs and
base64 attachments interleaved, so this returns matches from machine noise rather than from
anything anyone said. One run spent 8 of its 11 shell commands that way and surfaced hits
from inside base64 blobs. The skill now says to reach the transcripts through agsearch.
On a handoff they reach for git first, which is right as far as it goes. The skill now
says the transcript carries the reasoning and the options that were dropped, while git
carries what shipped, and that neither is the whole story alone.
Same prompts, before and after that revision:
Both tasks reached the same conclusion either way, so this is cost, not correctness. The
second task is the one worth having: the right answer is "there is no record of this", and
an agent that reports that instead of reconstructing something plausible is the whole point.
A bug this surfaced
A capped
readgiven search terms kept the closing turns and dropped the matched ones,which answers a question nobody asked. Matched turns now outrank the ending. On one long
session that took the kept matches from 8 to 27 inside the same budget.
Tests
141 total, 13 new: the frontmatter carries its triggers, every flag and subcommand the
skill teaches still exists in the CLI, the marketplace lists a plugin whose source
directory really contains a skill, its version is a version, and the elision keeps the
opening, the matches and the ending.