  Because of their state-of-the-art performance in computer vision, CNNs are
becoming increasingly popular in a variety of fields, including medicine.
However, as neural networks are black box function approximators, it is
difficult, if not impossible, for a medical expert to reason about their
output. This could potentially result in the expert distrusting the network
when he or she does not agree with its output. In such a case, explaining why
the CNN makes a certain decision becomes valuable information. In this paper,
we try to open the black box of the CNN by inspecting and visualizing the
learned feature maps, in the field of dermatology. We show that, to some
extent, CNNs focus on features similar to those used by dermatologists to make
a diagnosis. However, more research is required for fully explaining their
output.
