tokenize(text, {
        pair = null,
        add_special_tokens = false,
    } = {}) {
        return this._tokenize_helper(text, { pair, add_special_tokens }).tokens;
    }
Converts a string into a sequence of tokens.
@param {string} text The sequence to be encoded.
@param {Object} options An optional object containing the following properties:
@param {string} [options.pair] A second sequence to be encoded with the first.
@param {boolean} [options.add_special_tokens=false] Whether or not to add the special tokens associated with the corresponding model.
@returns {string[]} The list of tokens.