_tokenize_helper(text, {
        pair = null,
        add_special_tokens = false,
    } = {}) {
        const tokens = this._encode_text(text);
        const tokens2 = this._encode_text(pair);

        return this.post_processor
            ? this.post_processor(tokens, tokens2, { add_special_tokens })
            : { tokens: mergeArrays(tokens ?? [], tokens2 ?? []) };
    }
Internal helper function to tokenize a text, and optionally a pair of texts.
@param {string} text The text to tokenize.
@param {Object} options An optional object containing the following properties:
@param {string} [options.pair=null] The optional second text to tokenize.
@param {boolean} [options.add_special_tokens=false] Whether or not to add the special tokens associated with the corresponding model.
@returns {{tokens: string[], token_type_ids?: number[]}} An object containing the tokens and optionally the token type IDs.