To support Mistral Tekken tokenizer, the PR https://github.com/foundation-model-stack/foundation-model-stack/pull/434 added a new class of tokenizer in the `fms/utils/tokenizers.py` We need to add unit tests for : - Load config - Load tekken tokenizer - Encode/decode round-trip - Special token handling - vocab_size
To support Mistral Tekken tokenizer, the PR #434 added a new class of tokenizer in the
fms/utils/tokenizers.pyWe need to add unit tests for :