Tokenizer Byte Pair Encoding (BPE) The most popular is tiktoken. Resources https://huggingface.co/learn/nlp-course/en/chapter6/5