  Interpreting the prediction mechanism of complex models is currently one of
the most important tasks in the machine learning field, especially with layered
neural networks, which have achieved high predictive performance with various
practical data sets. To reveal the global structure of a trained neural network
in an interpretable way, a series of clustering methods have been proposed,
which decompose the units into clusters according to the similarity of their
inference roles. The main problems in these studies were that (1) we have no
prior knowledge about the optimal resolution for the decomposition, or the
appropriate number of clusters, and (2) there was no method with which to
acquire knowledge about whether the outputs of each cluster have a positive or
negative correlation with the input and output dimension values. In this paper,
to solve these problems, we propose a method for obtaining a hierarchical
modular representation of a layered neural network. The application of a
hierarchical clustering method to a trained network reveals a tree-structured
relationship among hidden layer units, based on their feature vectors defined
by their correlation with the input and output dimension values.
