  A core component present in many successful neural network architectures, is
an MLP block of two fully connected layers with a non-linear activation in
between. An intriguing phenomenon observed empirically, including in
transformer architectures, is that, after training, the activations in the
hidden layer of this MLP block tend to be extremely sparse on any given input.
Unlike traditional forms of sparsity, where there are neurons/weights which can
be deleted from the network, this form of {\em dynamic} activation sparsity
appears to be harder to exploit to get more efficient networks. Motivated by
this we initiate a formal study of PAC learnability of MLP layers that exhibit
activation sparsity. We present a variety of results showing that such classes
of functions do lead to provable computational and statistical advantages over
their non-sparse counterparts. Our hope is that a better theoretical
understanding of {\em sparsely activated} networks would lead to methods that
can exploit activation sparsity in practice.
