  Despite the impressive performance recently achieved by automatic speech
recognition (ASR), we observe two primary challenges that hinder its broader
applications: (1) The difficulty of introducing scalability into the model to
support more languages with limited training, inference, and storage overhead;
(2) The low-resource adaptation ability that enables effective low-resource
adaptation while avoiding over-fitting and catastrophic forgetting issues.
Inspired by recent findings, we hypothesize that we can address the above
challenges with modules widely shared across languages. To this end, we propose
an ASR framework, dubbed \METHODNS, that, \textit{for the first time},
simultaneously achieves strong multilingual scalability and low-resource
adaptation ability thanks to its modularize-then-assemble strategy.
Specifically, \METHOD learns a small set of generalizable sub-modules and
adaptively assembles them for different languages to reduce the multilingual
overhead and enable effective knowledge transfer for low-resource adaptation.
Extensive experiments and visualizations demonstrate that \METHOD can
effectively discover language similarity and improve multilingual and
low-resource ASR performance over state-of-the-art (SOTA) methods, e.g., under
multilingual-ASR, our framework achieves a 0.13$\sim$2.41 lower character error
rate (CER) with 30\% smaller inference overhead over SOTA solutions on
multilingual ASR and a comparable CER, with nearly 50 times fewer trainable
parameters over SOTA solutions on low-resource tuning, respectively.
