  Question answering (QA) systems are among the most important and rapidly
developing research topics in natural language processing (NLP). A reason,
therefore, is that a QA system allows humans to interact more naturally with a
machine, e.g., via a virtual assistant or search engine. In the last decades,
many QA systems have been proposed to address the requirements of different
question-answering tasks. Furthermore, many error scores have been introduced,
e.g., based on n-gram matching, word embeddings, or contextual embeddings to
measure the performance of a QA system. This survey attempts to provide a
systematic overview of the general framework of QA, QA paradigms, benchmark
datasets, and assessment techniques for a quantitative evaluation of QA
systems. The latter is particularly important because not only is the
construction of a QA system complex but also its evaluation. We hypothesize
that a reason, therefore, is that the quantitative formalization of human
judgment is an open problem.
