  Neural networks have revolutionized the field of data science, yielding
remarkable solutions in a data-driven manner. For instance, in the field of
mathematical imaging, they have surpassed traditional methods based on convex
regularization. However, a fundamental theory supporting the practical
applications is still in the early stages of development. We take a fresh look
at neural networks and examine them via nonlinear eigenvalue analysis. The
field of nonlinear spectral theory is still emerging, providing insights about
nonlinear operators and systems. In this paper we view a neural network as a
complex nonlinear operator and attempt to find its nonlinear eigenvectors. We
first discuss the existence of such eigenvectors and analyze the kernel of ReLU
networks. Then we study a nonlinear power method for generic nonlinear
operators. For proximal operators associated to absolutely one-homogeneous
convex regularization functionals, we can prove convergence of the method to an
eigenvector of the proximal operator. This motivates us to apply a nonlinear
method to networks which are trained to act similarly as a proximal operator.
In order to take the non-homogeneity of neural networks into account we define
a modified version of the power method.
  We perform extensive experiments for different proximal operators and on
various shallow and deep neural networks designed for image denoising. Proximal
eigenvectors will be used for geometric analysis of graphs, as clustering or
the computation of distance functions. For simple neural nets, we observe the
influence of training data on the eigenvectors. For state-of-the-art denoising
networks, we show that eigenvectors can be interpreted as (un)stable modes of
the network, when contaminated with noise or other degradations.
