  Existing metrics for evaluating the quality of automatically generated
questions such as BLEU, ROUGE, BERTScore, and BLEURT compare the reference and
predicted questions, providing a high score when there is a considerable
lexical overlap or semantic similarity between the candidate and the reference
questions. This approach has two major shortcomings. First, we need expensive
human-provided reference questions. Second, it penalises valid questions that
may not have high lexical or semantic similarity to the reference questions. In
this paper, we propose a new metric, RQUGE, based on the answerability of the
candidate question given the context. The metric consists of a
question-answering and a span scorer modules, using pre-trained models from
existing literature, thus it can be used without any further training. We
demonstrate that RQUGE has a higher correlation with human judgment without
relying on the reference question. Additionally, RQUGE is shown to be more
robust to several adversarial corruptions. Furthermore, we illustrate that we
can significantly improve the performance of QA models on out-of-domain
datasets by fine-tuning on synthetic data generated by a question generation
model and re-ranked by RQUGE.
