  Inspired from human cognition, machine learning systems are gradually
revealing advantages of sparser and more modular architectures. Recent work
demonstrates that not only do some modular architectures generalize well, but
they also lead to better out-of-distribution generalization, scaling
properties, learning speed, and interpretability. A key intuition behind the
success of such systems is that the data generating system for most real-world
settings is considered to consist of sparsely interacting parts, and endowing
models with similar inductive biases will be helpful. However, the field has
been lacking in a rigorous quantitative assessment of such systems because
these real-world data distributions are complex and unknown. In this work, we
provide a thorough assessment of common modular architectures, through the lens
of simple and known modular data distributions. We highlight the benefits of
modularity and sparsity and reveal insights on the challenges faced while
optimizing modular systems. In doing so, we propose evaluation metrics that
highlight the benefits of modularity, the regimes in which these benefits are
substantial, as well as the sub-optimality of current end-to-end learned
modular systems as opposed to their claimed potential.
