All pull requests

Pull requests

• Bump the actions group across 1 directory with 2 updates

  dependenciesgithub_actions
  #47·dependabot[bot] opened Oct 14, 2026Bot·

  ·

• Apply fuse_unk to each word instead of the whole token list

  #46·OsamaAnsar opened Oct 12, 2026·

  ·

• Split on all Unicode numeric characters in the Digits pre-tokenizer

  #45·OsamaAnsar opened Oct 12, 2026·

  ·

• Make the Whitespace pre-tokenizer Unicode-aware

  #44·OsamaAnsar opened Oct 12, 2026·

  ·

• Fix encoding of explicitly empty text pairs

  #43·agammann opened Oct 11, 2026·

  ·

• Truncate in encode, reserving room for special tokens

  #42·rich-murphey opened Oct 8, 2026·

  ·
  1

• Store CharTrie nodes in flat arrays instead of one map per node

  #38·alex-michaud opened Sep 20, 2026·

  ·

• Add offset tracking to Encoding

  #30·anidoesdev opened Jun 11, 2026·

  ·
  3

• feat: add offset mapping support for WordPiece tokenizers

  #17·dkhokhlov opened Dec 11, 2025·

  ·
  1

  1