convert_tokens_to_ids(tokens) {
        return tokens.map(t => this.tokens_to_ids.get(t) ?? this.unk_token_id);
    }
Converts a list of tokens into a list of token IDs.
@param {string[]} tokens The tokens to convert.
@returns {number[]} The converted token IDs.