Issues

• BertNormalizer does not pad CJK characters outside the BMP (tokenize_chinese_chars iterates UTF-16 code units)

  huggingface/tokenizers.js#49 · Filter by author trackoor opened on Nov 7, 2026

• Fix published types

  huggingface/tokenizers.js#48 · Filter by author s-h-a-d-o-w opened on Nov 2, 2026
  5

• Tokenizer truncation drops the trailing [SEP] (truncates after adding special tokens)

  huggingface/tokenizers.js#41 · Filter by author rich-murphey opened on Oct 7, 2026
  2

• Unigram tokenizer's CharTrie costs ~200+ MB RSS (~166 MB JS heap) for XLM-R-sized vocabs

  huggingface/tokenizers.js#37 · Filter by author tfriedel opened on Aug 2, 2026
  3

• tokenizers.js ignores byte_fallback for Unigram models

  enhancement
  huggingface/tokenizers.js#26 · Filter by author sodiumplus3 opened on Jun 4, 2026
  2

• Add offset mapping support for character-level token alignment

  huggingface/tokenizers.js#16 · Filter by author dkhokhlov opened on Dec 11, 2025
  1