  Modularity is a paradigm of machine translation with the potential of
bringing forth models that are large at training time and small during
inference. Within this field of study, modular approaches, and in particular
attention bridges, have been argued to improve the generalization capabilities
of models by fostering language-independent representations. In the present
paper, we study whether modularity affects translation quality; as well as how
well modular architectures generalize across different evaluation scenarios.
For a given computational budget, we find non-modular architectures to be
always comparable or preferable to all modular designs we study.
