  A major prerequisite for the application of machine learning models in
clinical decision making is trust and interpretability. Current explainability
studies in the neuroimaging community have mostly focused on explaining
individual decisions of trained models, e.g. obtained by a convolutional neural
network (CNN). Using attribution methods such as layer-wise relevance
propagation or SHAP heatmaps can be created that highlight which regions of an
input are more relevant for the decision than others. While this allows the
detection of potential data set biases and can be used as a guide for a human
expert, it does not allow an understanding of the underlying principles the
model has learned. In this study, we instead show, to the best of our
knowledge, for the first time results using feature visualization of
neuroimaging CNNs. Particularly, we have trained CNNs for different tasks
including sex classification and artificial lesion classification based on
structural magnetic resonance imaging (MRI) data. We have then iteratively
generated images that maximally activate specific neurons, in order to
visualize the patterns they respond to. To improve the visualizations we
compared several regularization strategies. The resulting images reveal the
learned concepts of the artificial lesions, including their shapes, but remain
hard to interpret for abstract features in the sex classification task.
