[sixgodgit]

Problem

Déjà Vu's tokenizer treated every English word of 3+ letters as a token. That included the, what, about, you, talked. Once a single English memory was imprinted, any unrelated English question came back "familiar". For example, sense("that quantum computing paper we talked about") said familiar because of about and that. The orchestrator then turned those words into [known entity] about recall candidates.

Change

scent() now skips a short list of English common words:

• function words
• contraction stems (don't → don)
• the most common everyday verbs
• the words recall questions are built from (remember, remind, talked, told, said, name, called…)

A capitalised phrase made only of such words ("What About") is skipped too. may stays a token, since it is also the month. Chinese tokenization is unchanged: the Chinese benchmark results and the tokenizer baseline are identical to v7.16.0.

No migration is needed. Words imprinted by older versions stay in Veil / Mist, but imprint and sense never produce them any more, so they are never probed.

Measured

benchmarks/dejavu_bench.py, new section E2:

• Background: 1,500 synthetic English memories written as natural sentences.
• Unrelated questions flagged familiar: 6/6 → 0/6.
• Related questions still familiar: 3/3. hunt() still finds the target: 3/3.

benchmarks/dejavu_english_real.py (new), on LongMemEval-S user turns with 10 unrelated questions:

Turns remembered	v7.16.0	v7.16.1
20	10/10 familiar	0/10
200	10/10	3/10
2,000	10/10	10/10

Not fixed (documented)

With a long history, ordinary content words like teacher and workshop have all been seen. familiar is true as soon as one content word has been seen, so those questions still come back familiar. This is how the design works, not a tokenizer bug. It is now point 6 of "What it is not" in docs/dejavu.md and its Chinese version, and it is in the README's known limitations.

Validation

• Full suite: 684 passed, 2 skipped, on Python 3.11 and 3.10.
• New tests in tests/test_dejavu.py:
  • Chinese tokenization is locked to the old result.
  • Common words are skipped.
  • Unrelated English questions are unfamiliar, and related ones are familiar.
  • Stopwords imprinted by older versions are never probed.

---

Generated by Claude Code