constructor(tokenizerJSON, tokenizerConfig) {
        super(tokenizerJSON, tokenizerConfig);

        this.languageRegex = /^__[a-z]{2,3}__$/;
        this.language_codes = this.special_tokens
            .filter(x => this.languageRegex.test(x))
            .map(x => x.slice(2, -2));
        this.lang_to_token = x => `__${x}__`;
    }
The M2M100Tokenizer class is used to tokenize text for M2M100 ("Many-to-Many") models.

M2M100 is a multilingual encoder-decoder (seq-to-seq) model trained for Many-to-Many
multilingual translation. It was introduced in this [paper](https://huggingface.co/papers/2010.11125)
and first released in [this](https://github.com/pytorch/fairseq/tree/master/examples/m2m_100) repository.

For a list of supported languages (along with their language codes),
@see {@link https://huggingface.co/facebook/m2m100_418M#languages-covered}