  We present a new approach to evaluate chord recognition systems on songs
which do not have full annotations. The principle is to use online chord
databases to generate high accurate "pseudo annotations" for these songs and
compute "pseudo accuracies" of test systems. Statistical models that model the
relationship between "pseudo accuracy" and real performance are then applied to
estimate test systems' performance. The approach goes beyond the existing
evaluation metrics, allowing us to carry out extensive analysis on chord
recognition systems, such as their generalizations to different genres. In the
experiments we applied this method to evaluate three state-of-the-art chord
recognition systems, of which the results verified its reliability.
