  Multilingual machine translation (MMT) benefits from cross-lingual transfer
but is a challenging multitask optimization problem. This is partly because
there is no clear framework to systematically learn language-specific
parameters. Self-supervised learning (SSL) approaches that leverage large
quantities of monolingual data (where parallel data is unavailable) have shown
promise by improving translation performance as complementary tasks to the MMT
task. However, jointly optimizing SSL and MMT tasks is even more challenging.
In this work, we first investigate how to utilize intra-distillation to learn
more *language-specific* parameters and then show the importance of these
language-specific parameters. Next, we propose a novel but simple SSL task,
concurrent denoising, that co-trains with the MMT task by concurrently
denoising monolingual data on both the encoder and decoder. Finally, we apply
intra-distillation to this co-training approach. Combining these two approaches
significantly improves MMT performance, outperforming three state-of-the-art
SSL methods by a large margin, e.g., 11.3\% and 3.7\% improvement on an
8-language and a 15-language benchmark compared with MASS, respectively
