[sixgodgit]

Problem

Chinese is cut into fixed-boundary two-character pieces. Half of the pieces in a sentence are function words or fragments that cross a word boundary. For example, "我们聊过的那家川菜馆" gave 我们 / 们聊 / 聊过 / 过的 / 的那 / 那家 / 家川 / 川菜 / 菜馆, and only the last three are about the restaurant.

On real Chinese chit-chat (LCCC), after only 20 remembered messages, 6 of 10 unrelated questions came back "familiar". The evidence was 一下, 什么, 我说, 的那, 过的…

Change

A Chinese piece no longer counts if either of these holds:

• It contains a common character: a particle (的 了 吗 呢 吧 啊 呀 嘛 哦 么), 我 / 你 / 您 / 们, 这 / 那, 是 or 在. These almost never belong to words worth remembering, but appear in most fragments.
• It is on a word list: common words, time deixis (上次, 之前, 最近, 上周…) and recall phrasing (记得, 聊过, 提过, 来着…).

A short run (3–4 characters) whose every adjacent pair is common gives nothing, e.g. 我们的 or 哈哈哈哈.

他 / 她 are deliberately not on the character list, so 吉他 and 其他 survive; 他们 / 他说 are on the word list instead. The tests also lock in that 成都, 过敏, 那不勒斯 and 吉他 still come through.

No migration is needed, same as v7.16.1.

Measured

benchmarks/dejavu_chinese_real.py (new; LCCC, MIT, Weibo chit-chat, 10 unrelated questions):

Utterances remembered	v7.16.1	v7.16.2
20	6/10 familiar	0/10
200	10/10	1/10 (工作)
2,000	10/10	3/10 (企鹅, 工作, 玻璃, 老师)

Other results:

• Section E of dejavu_bench.py (Chinese "familiar but unsearchable") is unchanged: 3/4 misses rescued, 0/3 noise. Section E2 (English) is also unchanged.
• Tokens per memory go down, so Veil fills more slowly. LCCC goes from 5.5 to 4.1 unique tokens per utterance, about 26,600 utterances to 1% false positives instead of about 20,000. The synthetic benchmark corpus goes from 37.7 to 35.2.
• The full Déjà Vu benchmark was re-run and the doc tables re-rendered. This machine differs from the earlier published runs, so latency is much lower (imprint p50 3.2 → 0.38 ms) because of the machine, not the code. On the same machine the v7.16.1 and v7.16.2 tokenizers measure imprint p50 0.37 vs 0.34 ms. The docs say this explicitly.

Not fixed

• Fragments made of two content characters (家川 in 那家川菜馆, 前跟 in 之前跟你) cannot be told apart without word segmentation, so they still count.
• Ordinary content words still accumulate with a long history. This is documented as point 6 of "What it is not".

Validation

• Full suite: 685 passed, 2 skipped, on Python 3.11 and 3.10.
• examples/dejavu_minimal.py runs; the README and doc examples match the real output.

---

Generated by Claude Code