Tokenizer

SentencePiece

SentencePiece implements subword units (e.g., byte-pair-encoding (BPE) and unigram language model.