  Multilingual pretrained representations generally rely on subword
segmentation algorithms to create a shared multilingual vocabulary. However,
standard heuristic algorithms often lead to sub-optimal segmentation,
especially for languages with limited amounts of data. In this paper, we take
two major steps towards alleviating this problem. First, we demonstrate
empirically that applying existing subword regularization methods(Kudo, 2018;
Provilkov et al., 2020) during fine-tuning of pre-trained multilingual
representations improves the effectiveness of cross-lingual transfer. Second,
to take full advantage of different possible input segmentations, we propose
Multi-view Subword Regularization (MVR), a method that enforces the consistency
between predictions of using inputs tokenized by the standard and probabilistic
segmentations. Results on the XTREME multilingual benchmark(Hu et al., 2020)
show that MVR brings consistent improvements of up to 2.5 points over using
standard segmentation algorithms.
