  Singing voice transcription converts recorded singing audio to musical
notation. Sound contamination (such as accompaniment) and lack of annotated
data make singing voice transcription an extremely difficult task. We take two
approaches to tackle the above challenges: 1) introducing multimodal learning
for singing voice transcription together with a new multimodal singing dataset,
N20EMv2, enhancing noise robustness by utilizing video information (lip
movements to predict the onset/offset of notes), and 2) adapting
self-supervised learning models from the speech domain to the singing voice
transcription task, significantly reducing annotated data requirements while
preserving pretrained features. We build a self-supervised learning based
audio-only singing voice transcription system, which not only outperforms
current state-of-the-art technologies as a strong baseline, but also
generalizes well to out-of-domain singing data. We then develop a
self-supervised learning based video-only singing voice transcription system
that detects note onsets and offsets with an accuracy of about 80\%. Finally,
based on the powerful acoustic and visual representations extracted by the
above two systems as well as the feature fusion design, we create an
audio-visual singing voice transcription system that improves the noise
robustness significantly under different acoustic environments compared to the
audio-only systems.
