  Web-crawled data provides a good source of parallel corpora for training
machine translation models. It is automatically obtained, but extremely noisy,
and recent work shows that neural machine translation systems are more
sensitive to noise than traditional statistical machine translation methods. In
this paper, we propose a novel approach to filter out noisy sentence pairs from
web-crawled corpora via pre-trained language models. We measure sentence
parallelism by leveraging the multilingual capability of BERT and use the
Generative Pre-training (GPT) language model as a domain filter to balance data
domains. We evaluate the proposed method on the WMT 2018 Parallel Corpus
Filtering shared task, and on our own web-crawled Japanese-Chinese parallel
corpus. Our method significantly outperforms baselines and achieves a new
state-of-the-art. In an unsupervised setting, our method achieves comparable
performance to the top-1 supervised method. We also evaluate on a web-crawled
Japanese-Chinese parallel corpus that we make publicly available.
