  Over the last decade, Convolutional Neural Networks (CNN) saw a tremendous
surge in performance. However, understanding what a network has learned still
proves to be a challenging task. To remedy this unsatisfactory situation, a
number of groups have recently proposed different methods to visualize the
learned models. In this work we suggest a general taxonomy to classify and
compare these methods, subdividing the literature into three main categories
and providing researchers with a terminology to base their works on.
Furthermore, we introduce the FeatureVis library for MatConvNet: an extendable,
easy to use open source library for visualizing CNNs. It contains
implementations from each of the three main classes of visualization methods
and serves as a useful tool for an enhanced understanding of the features
learned by intermediate layers, as well as for the analysis of why a network
might fail for certain examples.
