  Self-Supervised Learning (SSL) Automatic Speech Recognition (ASR) models have
shown great promise over Supervised Learning (SL) ones in low-resource
settings. However, the advantages of SSL are gradually weakened when the amount
of labeled data increases in many industrial applications. To further improve
the ASR performance when abundant labels are available, we first explore the
potential of combining SL and SSL ASR models via analyzing their
complementarity in recognition accuracy and optimization property. Then, we
propose a novel Optimal Transport based Fusion (OTF) method for SL and SSL
models without incurring extra computation cost in inference. Specifically,
optimal transport is adopted to softly align the layer-wise weights to unify
the two different networks into a single one. Experimental results on the
public 1k-hour English LibriSpeech dataset and our in-house 2.6k-hour Chinese
dataset show that OTF largely outperforms the individual models with lower
error rates.
