  The power law has been observed in the degree distributions of many
biological neural networks. Sparse deep neural networks, which learn an
economical representation from the data, resemble biological neural networks in
many ways. In this paper, we study if these artificial networks also exhibit
properties of the power law. Experimental results on two popular deep learning
models, namely, multilayer perceptrons and convolutional neural networks, are
affirmative. The power law is also naturally related to preferential
attachment. To study the dynamical properties of deep networks in continual
learning, we propose an internal preferential attachment model to explain how
the network topology evolves. Experimental results show that with the arrival
of a new task, the new connections made follow this preferential attachment
process.
