  Biomedical image analysis algorithm validation depends on high-quality
annotation of reference datasets, for which labeling instructions are key.
Despite their importance, their optimization remains largely unexplored. Here,
we present the first systematic study of labeling instructions and their impact
on annotation quality in the field. Through comprehensive examination of
professional practice and international competitions registered at the MICCAI
Society, we uncovered a discrepancy between annotators' needs for labeling
instructions and their current quality and availability. Based on an analysis
of 14,040 images annotated by 156 annotators from four professional companies
and 708 Amazon Mechanical Turk (MTurk) crowdworkers using instructions with
different information density levels, we further found that including exemplary
images significantly boosts annotation performance compared to text-only
descriptions, while solely extending text descriptions does not. Finally,
professional annotators constantly outperform MTurk crowdworkers. Our study
raises awareness for the need of quality standards in biomedical image analysis
labeling instructions.
