  We argue the Fisher information matrix (FIM) of one hidden layer networks
with the ReLU activation function. For a network, let $W$ denote the $d \times
p$ weight matrix from the $d$-dimensional input to the hidden layer consisting
of $p$ neurons, and $v$ the $p$-dimensional weight vector from the hidden layer
to the scalar output. We focus on the FIM of $v$, which we denote as $I$. Under
certain conditions, we characterize the first three clusters of eigenvalues and
eigenvectors of the FIM. Specifically, we show that 1) Since $I$ is
non-negative owing to the ReLU, the first eigenvalue is the Perron-Frobenius
eigenvalue. 2) For the cluster of the next maximum values, the eigenspace is
spanned by the row vectors of $W$. 3) The direct sum of the eigenspace of the
first eigenvalue and that of the third cluster is spanned by the set of all the
vectors obtained as the Hadamard product of any pair of the row vectors of $W$.
We confirmed by numerical calculation that the above is approximately correct
when the number of hidden nodes is about 10000.
