  We introduce AVA, an automatic evaluation approach for Question Answering,
which given a set of questions associated with Gold Standard answers, can
estimate system Accuracy. AVA uses Transformer-based language models to encode
question, answer, and reference text. This allows for effectively measuring the
similarity between the reference and an automatic answer, biased towards the
question semantics. To design, train and test AVA, we built multiple large
training, development, and test sets on both public and industrial benchmarks.
Our innovative solutions achieve up to 74.7% in F1 score in predicting human
judgement for single answers. Additionally, AVA can be used to evaluate the
overall system Accuracy with an RMSE, ranging from 0.02 to 0.09, depending on
the availability of multiple references.
