  Multilingual Machine Translation promises to improve translation quality
between non-English languages. This is advantageous for several reasons, namely
lower latency (no need to translate twice), and reduced error cascades (e.g.,
avoiding losing gender and formality information when translating through
English). On the downside, adding more languages reduces model capacity per
language, which is usually countered by increasing the overall model size,
making training harder and inference slower. In this work, we introduce
Language-Specific Transformer Layers (LSLs), which allow us to increase model
capacity, while keeping the amount of computation and the number of parameters
used in the forward pass constant. The key idea is to have some layers of the
encoder be source or target language-specific, while keeping the remaining
layers shared. We study the best way to place these layers using a neural
architecture search inspired approach, and achieve an improvement of 1.3 chrF
(1.5 spBLEU) points over not using LSLs on a separate decoder architecture, and
1.9 chrF (2.2 spBLEU) on a shared decoder one.
