  Deep Learning (DL) , a variant of the neural network algorithms originally
proposed in the 1980s, has made surprising progress in Artificial Intelligence
(AI), ranging from language translation, protein folding, autonomous cars, and
more recently human-like language models (CHATbots), all that seemed
intractable until very recently. Despite the growing use of Deep Learning (DL)
networks, little is actually understood about the learning mechanisms and
representations that makes these networks effective across such a diverse range
of applications. Part of the answer must be the huge scale of the architecture
and of course the large scale of the data, since not much has changed since
1987. But the nature of deep learned representations remain largely unknown.
Unfortunately training sets with millions or billions of tokens have unknown
combinatorics and Networks with millions or billions of hidden units cannot
easily be visualized and their mechanisms cannot be easily revealed. In this
paper, we explore these questions with a large (1.24M weights; VGG) DL in a
novel high density sample task (5 unique tokens with at minimum 500 exemplars
per token) which allows us to more carefully follow the emergence of category
structure and feature construction. We use various visualization methods for
following the emergence of the classification and the development of the
coupling of feature detectors and structures that provide a type of graphical
bootstrapping, From these results we harvest some basic observations of the
learning dynamics of DL and propose a new theory of complex feature
construction based on our results.
