  We prove the converse of the universal approximation theorem, i.e. a neural
network (NN) encoding theorem which shows that for every stably converged NN of
continuous activation functions, its weight matrix actually encodes a
continuous function that approximates its training dataset to within a finite
margin of error over a bounded domain. We further show that using the
Eckart-Young theorem for truncated singular value decomposition of the weight
matrix for every NN layer, we can illuminate the nature of the latent space
manifold of the training dataset encoded and represented by every NN layer, and
the geometric nature of the mathematical operations performed by each NN layer.
Our results have implications for understanding how NNs break the curse of
dimensionality by harnessing memory capacity for expressivity, and that the two
are complementary. This Layer Matrix Decomposition (LMD) further suggests a
close relationship between eigen-decomposition of NN layers and the latest
advances in conceptualizations of Hopfield networks and Transformer NN models.
