  Recent advances in machine learning models have greatly increased the
performance of automated methods in medical image analysis. However, the
internal functioning of such models is largely hidden, which hinders their
integration in clinical practice. Explainability and trust are viewed as
important aspects of modern methods, for the latter's widespread use in
clinical communities. As such, validation of machine learning models represents
an important aspect and yet, most methods are only validated in a limited way.
In this work, we focus on providing a richer and more appropriate validation
approach for highly powerful Visual Question Answering (VQA) algorithms. To
better understand the performance of these methods, which answer arbitrary
questions related to images, this work focuses on an automatic visual Turing
test (VTT). That is, we propose an automatic adaptive questioning method, that
aims to expose the reasoning behavior of a VQA algorithm. Specifically, we
introduce a reinforcement learning (RL) agent that observes the history of
previously asked questions, and uses it to select the next question to pose. We
demonstrate our approach in the context of evaluating algorithms that
automatically answer questions related to diabetic macular edema (DME) grading.
The experiments show that such an agent has similar behavior to a clinician,
whereby asking questions that are relevant to key clinical concepts.
