pre_tokenize_text(text, options) {
        if (this.pattern === null) {
            return [];
        }

        if (this.config.invert) {
            return text.match(this.pattern) || [];
        } else if (this.config.behavior?.toLowerCase() === 'removed') {
            return text.split(this.pattern).filter(x => x);
        } else {
            return regexSplit(text, this.pattern);
        }
    }
Tokenizes text by splitting it using the given pattern.
@param {string} text The text to tokenize.
@param {Object} [options] Additional options for the pre-tokenization logic.
@returns {string[]} An array of tokens.