Fix: remove spurious spaces between CJK characters when compressing (#131)

[Beytab]

Summary

Fixes #131. When the input is Chinese (or other CJK text), the compressed output contains many spurious spaces between characters.

Root cause

In PromptCompressor, tokenizer.convert_tokens_to_string(keep_words) joins words with ASCII spaces — an English word-boundary convention. CJK scripts have no word delimiter, so this inserts spurious spaces between CJK characters (and around CJK punctuation).

Fix

Add a remove_space_between_cjk helper in llmlingua/utils.py that strips an ASCII space sitting between two CJK characters, and apply it to keep_str in prompt_compressor.py.

The regex covers CJK ideographs (incl. Extension A / compatibility), Japanese kana, Korean hangul, and CJK/fullwidth punctuation blocks, so spaces around CJK punctuation (e.g. ，。：、！？) are removed too. It only removes a space when both neighbors are CJK, so English text and mixed CJK/ASCII spacing are left intact.

Testing

Adds tests/test_llmlingua2_chinese.py covering the CJK space-removal behavior.

[Beytab]

Beytab please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

[Beytab]

@microsoft-github-policy-service agree