  Recent machine translation algorithms mainly rely on parallel corpora.
However, since the availability of parallel corpora remains limited, only some
resource-rich language pairs can benefit from them. We constructed a parallel
corpus for English-Japanese, for which the amount of publicly available
parallel corpora is still limited. We constructed the parallel corpus by
broadly crawling the web and automatically aligning parallel sentences. Our
collected corpus, called JParaCrawl, amassed over 8.7 million sentence pairs.
We show how it includes a broader range of domains and how a neural machine
translation model trained with it works as a good pre-trained model for
fine-tuning specific domains. The pre-training and fine-tuning approaches
achieved or surpassed performance comparable to model training from the initial
state and reduced the training time. Additionally, we trained the model with an
in-domain dataset and JParaCrawl to show how we achieved the best performance
with them. JParaCrawl and the pre-trained models are freely available online
for research purposes.
