  Analyzing the inner mechanisms of deep neural networks is a fundamental task
in machine learning. Existing work provides limited analysis or it depends on
local theories, such as fixed-point analysis. In contrast, we propose to
analyze trained neural networks using an operator theoretic approach which is
rooted in Koopman theory, the Koopman Analysis of Neural Networks (KANN). Key
to our method is the Koopman operator, which is a linear object that globally
represents the dominant behavior of the network dynamics. The linearity of the
Koopman operator facilitates analysis via its eigenvectors and eigenvalues. Our
method reveals that the latter eigendecomposition holds semantic information
related to the neural network inner workings. For instance, the eigenvectors
highlight positive and negative n-grams in the sentiments analysis task;
similarly, the eigenvectors capture the salient features of healthy heart beat
signals in the ECG classification problem.
