  With the increasing availability of new image registration approaches, an
unbiased evaluation is becoming more needed so that clinicians can choose the
most suitable approaches for their applications. Current evaluations typically
use landmarks in manually annotated datasets. As a result, the quality of
annotations is crucial for unbiased comparisons. Even though most data
providers claim to have quality control over their datasets, an objective
third-party screening can be reassuring for intended users. In this study, we
use the variogram to screen the manually annotated landmarks in two datasets
used to benchmark registration in image-guided neurosurgeries. The variogram
provides an intuitive 2D representation of the spatial characteristics of
annotated landmarks. Using variograms, we identified potentially problematic
cases and had them examined by experienced radiologists. We found that (1) a
small number of annotations may have fiducial localization errors; (2) the
landmark distribution for some cases is not ideal to offer fair comparisons. If
unresolved, both findings could incur bias in registration evaluation.
