  Multilingual pretraining and fine-tuning have remarkably succeeded in various
natural language processing tasks. Transferring representations from one
language to another is especially crucial for cross-lingual learning. One can
expect machine translation objectives to be well suited to fostering such
capabilities, as they involve the explicit alignment of semantically equivalent
sentences from different languages. This paper investigates the potential
benefits of employing machine translation as a continued training objective to
enhance language representation learning, bridging multilingual pretraining and
cross-lingual applications. We study this question through two lenses: a
quantitative evaluation of the performance of existing models and an analysis
of their latent representations. Our results show that, contrary to
expectations, machine translation as the continued training fails to enhance
cross-lingual representation learning in multiple cross-lingual natural
language understanding tasks. We conclude that explicit sentence-level
alignment in the cross-lingual scenario is detrimental to cross-lingual
transfer pretraining, which has important implications for future cross-lingual
transfer studies. We furthermore provide evidence through similarity measures
and investigation of parameters that this lack of positive influence is due to
output separability -- which we argue is of use for machine translation but
detrimental elsewhere.
