  Recent studies have shown the advantages of evaluating NLG systems using
pairwise comparisons as opposed to direct assessment. Given $k$ systems, a
naive approach for identifying the top-ranked system would be to uniformly
obtain pairwise comparisons from all ${k \choose 2}$ pairs of systems. However,
this can be very expensive as the number of human annotations required would
grow quadratically with $k$. In this work, we introduce Active Evaluation, a
framework to efficiently identify the top-ranked system by actively choosing
system pairs for comparison using dueling bandit algorithms. We perform
extensive experiments with 13 dueling bandits algorithms on 13 NLG evaluation
datasets spanning 5 tasks and show that the number of human annotations can be
reduced by 80%. To further reduce the number of human annotations, we propose
model-based dueling bandit algorithms which combine automatic evaluation
metrics with human evaluations. Specifically, we eliminate sub-optimal systems
even before the human annotation process and perform human evaluations only on
test examples where the automatic metric is highly uncertain. This reduces the
number of human annotations required further by 89%. In effect, we show that
identifying the top-ranked system requires only a few hundred human
annotations, which grow linearly with $k$. Lastly, we provide practical
recommendations and best practices to identify the top-ranked system
efficiently. Our code has been made publicly available at
https://github.com/akashkm99/duelnlg
