  The recently proposed Conformer architecture which combines convolution with
attention to capture both local and global dependencies has become the
\textit{de facto} backbone model for Automatic Speech Recognition~(ASR).
Inherited from the Natural Language Processing (NLP) tasks, the architecture
takes Layer Normalization~(LN) as a default normalization technique. However,
through a series of systematic studies, we find that LN might take 10\% of the
inference time despite that it only contributes to 0.1\% of the FLOPs. This
motivates us to replace LN with other normalization techniques, e.g., Batch
Normalization~(BN), to speed up inference with the help of operator fusion
methods and the avoidance of calculating the mean and variance statistics
during inference. After examining several plain attempts which directly remove
all LN layers or replace them with BN in the same place, we find that the
divergence issue is mainly caused by the unstable layer output. We therefore
propose to append a BN layer to each linear or convolution layer where
stabilized training results are observed. We also propose to simplify the
activations in Conformer, such as Swish and GLU, by replacing them with ReLU.
All these exchanged modules can be fused into the weights of the adjacent
linear/convolution layers and hence have zero inference cost. Therefore, we
name it FusionFormer. Our experiments indicate that FusionFormer is as
effective as the LN-based Conformer and is about 10\% faster.
