  We present a skin condition classification methodology based on a sequential
pipeline of a pre-trained Convolutional Neural Network (CNN) and a Question
Answering (QA) model. This method enables us to not only increase the
classification confidence and accuracy of the deployed CNN system, but also
enables the emulation of the conventional approach of doctors asking the
relevant questions in refining the ultimate diagnosis and differential. By
combining the CNN output in the form of classification probabilities as a prior
to the QA model and the image textual description, we greedily ask the best
symptom that maximizes the information gain over symptoms. We demonstrate that
combining the QA model with the CNN increases the accuracy up to 10% as
compared to the CNN alone, and more than 30% as compared to the QA model alone.
