  We describe a novel method for efficiently eliciting scalar annotations for
dataset construction and system quality estimation by human judgments. We
contrast direct assessment (annotators assign scores to items directly), online
pairwise ranking aggregation (scores derive from annotator comparison of
items), and a hybrid approach (EASL: Efficient Annotation of Scalar Labels)
proposed here. Our proposal leads to increased correlation with ground truth,
at far greater annotator efficiency, suggesting this strategy as an improved
mechanism for dataset creation and manual system evaluation.
