  This paper studies the problem of automatically generating piano score
following videos given an audio recording and raw sheet music images. Whereas
previous works focus on synthetic sheet music where the data has been cleaned
and preprocessed, we instead focus on developing a system that can cope with
the messiness of raw, unprocessed sheet music PDFs from IMSLP. We investigate
how well existing systems cope with real scanned sheet music, filler pages and
unrelated pieces or movements, and discontinuities due to jumps and repeats. We
find that a significant bottleneck in system performance is handling jumps and
repeats correctly. In particular, we find that a previously proposed Jump DTW
algorithm does not perform robustly when jump locations are unknown a priori.
We propose a novel alignment algorithm called Hierarchical DTW that can handle
jumps and repeats even when jump locations are not known. It first performs
alignment at the feature level on each sheet music line, and then performs a
second alignment at the segment level. By operating at the segment level, it is
able to encode domain knowledge about how likely a particular jump is. Through
carefully controlled experiments on unprocessed sheet music PDFs from IMSLP, we
show that Hierarachical DTW significantly outperforms Jump DTW in handling
various types of jumps.
