detokenizer

Here are 5 public repositories matching this topic...

Thot toolkit for statistical machine translation

De-Tokenize is a Python package which provides fast, accurate structuring of tokens back to original sentence form

python nlp python3 detokenizer detokenize littinrajan

Pre/post processing utilities for the Thot translation toolkit.

tokenizer detokenizer recaser

Demonstrates how to convert a tokenizer into OpenVINO IR model and how to integrate it into an NLP model.

Convert Microsoft QuickBASIC 4.5 binary .BAS files (the 0xFC format) back to readable BASIC source text, without DOS

python basic dos file-format retrocomputing msdos qbasic detokenizer quickbasic qb45 tokenized

To associate your repository with the detokenizer topic, visit your repo's landing page and select "manage topics."