  Synapses in real neural circuits can take discrete values, including zero
(silent or potential) synapses. The computational role of zero synapses in
unsupervised feature learning of unlabeled noisy data is still unclear, thus it
is important to understand how the sparseness of synaptic activity is shaped
during learning and its relationship with receptive field formation. Here, we
formulate this kind of sparse feature learning by a statistical mechanics
approach. We find that learning decreases the fraction of zero synapses, and
when the fraction decreases rapidly around a critical data size, an
intrinsically structured receptive field starts to develop. Further increasing
the data size refines the receptive field, while a very small fraction of zero
synapses remain to act as contour detectors. This phenomenon is discovered not
only in learning a handwritten digits dataset, but also in learning retinal
neural activity measured in a natural-movie-stimuli experiment.
