  Multilingual neural machine translation (MNMT) aims to build a unified model
for many language directions. Existing monolithic models for MNMT encounter two
challenges: parameter interference among languages and inefficient inference
for large models. In this paper, we revisit the classic multi-way structures
and develop a detachable model by assigning each language (or group of
languages) to an individual branch that supports plug-and-play training and
inference. To address the needs of learning representations for all languages
in a unified space, we propose a novel efficient training recipe, upon which we
build an effective detachable model, Lego-MT. For a fair comparison, we collect
data from OPUS and build a translation benchmark covering 433 languages and
1.3B parallel data. Experiments show that Lego-MT with 1.2B parameters brings
an average gain of 3.2 spBLEU. It even outperforms M2M-100 with 12B parameters.
The proposed training recipe brings a 28.2$\times$ speedup over the
conventional multi-way training method.\footnote{
\url{https://github.com/CONE-MT/Lego-MT}.}
