  This paper provides a theoretical framework on the solution of feedforward
ReLU networks for interpolations, in terms of what is called an interpolation
matrix, which is the summary, extension and generalization of our three
preceding works, with the expectation that the solution of engineering could be
included in this framework and finally understood. To three-layer networks, we
classify different kinds of solutions and model them in a normalized form; the
solution finding is investigated by three dimensions, including data, networks
and the training; the mechanism of a type of overparameterization solution is
interpreted. To deep-layer networks, we present a general result called
sparse-matrix principle, which could describe some basic behavior of deep
layers and explain the phenomenon of the sparse-activation mode that appears in
engineering applications associated with brain science; an advantage of deep
layers compared to shallower ones is manifested in this principle. As
applications, a general solution of deep neural networks for classifications is
constructed by that principle; and we also use the principle to study the
data-disentangling property of encoders. Analogous to the three-layer case, the
solution of deep layers is also explored through several dimensions. The
mechanism of multi-output neural networks is explained from the perspective of
interpolation matrices.
