  Piano fingering -- knowing which finger to use to play each note in a musical
piece, is a hard and important skill to master when learning to play the piano.
While some sheet music is available with expert-annotated fingering
information, most pieces lack this information, and people often resort to
learning the fingering from demonstrations in online videos. We consider the AI
task of automating the extraction of fingering information from videos. This is
a non-trivial task as fingers are often occluded by other fingers, and it is
often not clear from the video which of the keys were pressed, requiring the
synchronization of hand position information and knowledge about the notes that
were played. We show how to perform this task with high-accuracy using a
combination of deep-learning modules, including a GAN-based approach for
fine-tuning on out-of-domain data. We extract the fingering information with an
f1 score of 97\%. We run the resulting system on 90 videos, resulting in
high-quality piano fingering information of 150K notes, the largest available
dataset of piano-fingering to date.
