  A theory of neural networks (NNs) built upon collective variables would
provide scientists with the tools to better understand the learning process at
every stage. In this work, we introduce two such variables, the entropy and the
trace of the empirical neural tangent kernel (NTK) built on the training data
passed to the model. We empirically analyze the NN performance in the context
of these variables and find that there exists correlation between the starting
entropy, the trace of the NTK, and the generalization of the model computed
after training is complete. This framework is then applied to the problem of
optimal data selection for the training of NNs. To this end, random network
distillation (RND) is used as a means of selecting training data which is then
compared with random selection of data. It is shown that not only does RND
select data-sets capable of outperforming random selection, but that the
collective variables associated with the RND data-sets are larger than those of
the randomly selected sets. The results of this investigation provide a stable
ground from which the selection of data for NN training can be driven by this
phenomenological framework.
