pre_tokenize(text, options) {
        return (Array.isArray(text)
            ? text.map(x => this.pre_tokenize_text(x, options))
            : this.pre_tokenize_text(text, options)
        ).flat();
    }
Tokenizes the given text into pre-tokens.
@param {string|string[]} text The text or array of texts to pre-tokenize.
@param {Object} [options] Additional options for the pre-tokenization logic.
@returns {string[]} An array of pre-tokens.