  Conformer has proven to be effective in many speech processing tasks. It
combines the benefits of extracting local dependencies using convolutions and
global dependencies using self-attention. Inspired by this, we propose a more
flexible, interpretable and customizable encoder alternative, Branchformer,
with parallel branches for modeling various ranged dependencies in end-to-end
speech processing. In each encoder layer, one branch employs self-attention or
its variant to capture long-range dependencies, while the other branch utilizes
an MLP module with convolutional gating (cgMLP) to extract local relationships.
We conduct experiments on several speech recognition and spoken language
understanding benchmarks. Results show that our model outperforms both
Transformer and cgMLP. It also matches with or outperforms state-of-the-art
results achieved by Conformer. Furthermore, we show various strategies to
reduce computation thanks to the two-branch architecture, including the ability
to have variable inference complexity in a single trained model. The weights
learned for merging branches indicate how local and global dependencies are
utilized in different layers, which benefits model designing.
