  In this study, we propose a simple and effective preprocessing method for
subword segmentation based on a data compression algorithm. Compression-based
subword segmentation has recently attracted significant attention as a
preprocessing method for training data in Neural Machine Translation. Among
them, BPE/BPE-dropout is one of the fastest and most effective method compared
to conventional approaches. However, compression-based approach has a drawback
in that generating multiple segmentations is difficult due to the determinism.
To overcome this difficulty, we focus on a probabilistic string algorithm,
called locally-consistent parsing (LCP), that has been applied to achieve
optimum compression. Employing the probabilistic mechanism of LCP, we propose
LCP-dropout for multiple subword segmentation that improves BPE/BPE-dropout,
and show that it outperforms various baselines in learning from especially
small training data.
