  This paper proposes a method to optimize tokenization for the performance
improvement of already trained downstream models. Our method generates
tokenization results attaining lower loss values of a given downstream model on
the training data for restricting vocabularies and trains a tokenizer
reproducing the tokenization results. Therefore, our method can be applied to
variety of tokenization methods, while existing work cannot due to the
simultaneous learning of the tokenizer and the downstream model. This paper
proposes an example of the BiLSTM-based tokenizer with vocabulary restriction,
which can capture wider contextual information for the tokenization process
than non-neural-based tokenization methods used in existing work. Experimental
results on text classification in Japanese, Chinese, and English text
classification tasks show that the proposed method improves performance
compared to the existing methods for tokenization optimization.
