  Automatic speech recognition (ASR) has progressed significantly in recent
years due to the emergence of large-scale datasets and the self-supervised
learning (SSL) paradigm. However, as its counterpart problem in the singing
domain, the development of automatic lyric transcription (ALT) suffers from
limited data and degraded intelligibility of sung lyrics. To fill in the
performance gap between ALT and ASR, we attempt to exploit the similarities
between speech and singing. In this work, we propose a transfer-learning-based
ALT solution that takes advantage of these similarities by adapting wav2vec
2.0, an SSL ASR model, to the singing domain. We maximize the effectiveness of
transfer learning by exploring the influence of different transfer starting
points. We further enhance the performance by extending the original CTC model
to a hybrid CTC/attention model. Our method surpasses previous approaches by a
large margin on various ALT benchmark datasets. Further experiments show that,
with even a tiny proportion of training data, our method still achieves
competitive performance.
