  Adapters are an efficient, composable alternative to full fine-tuning of
pre-trained models and help scale the deployment of large ASR models to many
tasks. In practice, a task ID is commonly prepended to the input during
inference to route to single-task adapters for the specified task. However, one
major limitation of this approach is that the task ID may not be known during
inference, rendering it unsuitable for most multi-task settings. To address
this, we propose three novel task-ID-free methods to combine single-task
adapters in multi-task ASR and investigate two learning algorithms for
training. We evaluate our methods on 10 test sets from 4 diverse ASR tasks and
show that our methods are non-destructive and parameter-efficient. While only
updating 17% of the model parameters, our methods can achieve an 8% mean WER
improvement relative to full fine-tuning and are on-par with task-ID adapter
routing.
