Skip to content

fix: keep earlier answers and memory after a source is deleted - #170

Open
devhims wants to merge 4 commits into
mainfrom
fix/history-survives-deletion
Open

devhims wants to merge 4 commits into
mainfrom
fix/history-survives-deletion

Conversation

@devhims

@devhims devhims commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Problem / Motivation

Deleting session evidence made the agent forget its own earlier answers. For every answer that used the deleted source:

Where Before
Dashboard Answer visible, citations shown as [source deleted]
Agent prompt (recent turns) Answer replaced with "[Historical answer omitted because its supporting evidence was deleted.]"
History search (search_context, read_session_history) Answer removed

"Clear saved data" did this to every answer in the session. The agent kept the user's questions but lost its own replies, so a follow-up like "does that still fit my budget?" lost its context. The deletion dialog says "Conversation messages remain", which was true for the user but not for the agent.

Deleting a source should remove the source, not the conversation. Earlier answers are history: useful context, never evidence.

What changed

  • Deletion keeps answers in history. The source, its excerpts and the citations to it are still removed; citations show [source deleted]. The redacted answer stays in the agent's prompt and in history search.

  • History is never a source. New HISTORY_ATTRIBUTION_GUIDANCE in the answer model's instructions: earlier answers, including ones showing [source deleted], are conversation history. When any part of an answer relies on something stated only earlier, the model says so plainly ("Based on our earlier conversation, ...") and never cites it or presents it as verified evidence. This complements the existing guidance that history is untrusted context.

  • Existing sessions recover. History indexing is versioned (HISTORY_REVISION), so the next sync re-indexes every run once and restores answers removed under the old rule.

  • Dialog copy: "Conversation messages remain, and the agent can still refer to them as earlier conversation."

  • Memory is kept as recorded. Deleting one source used to delete every memory entry that cited any of its excerpts, even an entry that also cited other sources. Memory records what was answered for the user's question, so deletion no longer rewrites or erases it. An affected entry keeps its text and evidenceIds and lists the lost citations in deletedEvidenceIds. The dashboard shows "N deleted, now unverified", and the answer model treats that part as unverified earlier conversation and says so when using it. A later rewrite of the topic keeps the mark, since the rewritten text can carry the deleted source's facts forward; forgetting the memory clears it.

Unchanged:

  • New memory cannot cite deleted evidence, and no memory is built from an answer whose evidence was deleted.
  • "Clear saved data" explicitly includes memory in its dialog and still removes it.

Attribution relies on the model following its instructions; there is no application check that an answer labels history it used.

Tests

All 1,592 platform unit tests, 352 runtime/session integration tests and 104 web tests pass, plus the dashboard deletion E2E test. Type checks and the docs check pass. New or updated tests cover:

  • a follow-up run's prompt still containing the earlier answer after its source is deleted, with [source deleted] and without the deleted source's text
  • re-indexing restoring an answer removed under the old revision format
  • deletion keeping the redacted answer searchable while the source text stays gone
  • the history backfill now including answers whose evidence was deleted
  • the answer model's prompt containing the attribution rule, including the rule for memory with deleted sources
  • a memory rewrite keeping the deleted-source mark, and forgetting the memory clearing it
  • the deletion dialogs: source deletion keeps memory marked unverified, clearing all saved data removes memory
  • a memory citing two sources keeping its text and live citation when one is deleted, without duplicate marks on repeated deletion
  • memory, dashboard inventory and memory search keeping findings after source deletion, while a late write citing deleted evidence is still rejected

Relationship to #169

Independent. #169's ID-based coverage check stays correct either way; its deleted-answers edge case stops arising once this merges. Both edit SESSION_EVIDENCE.md in different paragraphs.

Compatibility and deployment

No migration. The first run in each session after deploy re-indexes that session's history once.

Deleting session evidence removed every affected answer from the agent's
view: the prompt replaced it with "[Historical answer omitted because
its supporting evidence was deleted.]" and history search dropped it.
"Clear saved data" did this to every answer in the session. The agent
kept the user's questions but forgot its own replies, although the
deletion dialog says conversation messages remain.

Deleting a source now removes the source, its excerpts and the citations
to it, shown as "[source deleted]", while the answer stays in the
prompt's conversation history and in history search. The answer model
is told that earlier conversation is history, never a source: when it
relies on something stated only earlier, it says so ("Based on our
earlier conversation, ...") and does not cite it.

The history indexing rule is versioned so the next sync re-indexes every
run once, restoring answers that were removed under the old rule. Memory
is still not built from answers whose evidence was deleted.
@vercel

vercel Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
video2ctx-web Ready Ready Preview Oct 8, 2026 6:20am UTC

Deleting one source deleted every memory entry that cited any of its
excerpts, even an entry that also cited other sources. Memory records
what was answered for the user's question, so deleting a source should
not rewrite or erase it.

An entry that cited the deleted source now keeps its text and
evidenceIds unchanged and lists the lost citations in
deletedEvidenceIds. The dashboard shows them as deleted and unverified,
and the answer model is told to treat that part as unverified earlier
conversation and say so when using it. New memory still cannot cite
deleted evidence. "Clear saved data" explicitly includes memory and
still removes it.
@devhims devhims changed the title fix: keep earlier answers in agent history after a source is deleted fix: keep earlier answers and memory after a source is deleted Oct 8, 2026
When the memory updater rewrote a topic, it stored a fresh entry and the
deletedEvidenceIds mark disappeared. The rewritten text can carry facts
from the deleted source forward under live citations, so they would look
verified again. A rewrite now keeps the existing mark; forgetting the
memory clears it.

Also updates the dashboard E2E test for the clarified deletion dialogs:
deleting one source keeps memory with its citation marked deleted and
unverified, and clearing all saved data still removes memory.

This branch was successfully deployed

1 active deployment
Preview — ee4b110a Deployed Oct 8, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant