  Automatic piano transcription models are typically evaluated using simple
frame- or note-wise information retrieval (IR) metrics. Such benchmark metrics
do not provide insights into the transcription quality of specific musical
aspects such as articulation, dynamics, or rhythmic precision of the output,
which are essential in the context of expressive performance analysis.
Furthermore, in recent years, MAESTRO has become the de-facto training and
evaluation dataset for such models. However, inference performance has been
observed to deteriorate substantially when applied on out-of-distribution data,
thereby questioning the suitability and reliability of transcribed outputs from
such models for specific MIR tasks. In this work, we investigate the
performance of three state-of-the-art piano transcription models in two
experiments. In the first one, we propose a variety of musically informed
evaluation metrics which, in contrast to the IR metrics, offer more detailed
insight into the musical quality of the transcriptions. In the second
experiment, we compare inference performance on real-world and perturbed audio
recordings, and highlight musical dimensions which our metrics can help
explain. Our experimental results highlight the weaknesses of existing piano
transcription metrics and contribute to a more musically sound error analysis
of transcription outputs.
