  Leveraging sparse networks to connect successive layers in deep neural
networks has recently been shown to provide benefits to large scale
state-of-the-art models. However, network connectivity also plays a significant
role on the learning performance of shallow networks, such as the classic
Restricted Boltzmann Machines (RBM). Efficiently finding sparse connectivity
patterns that improve the learning performance of shallow networks is a
fundamental problem. While recent principled approaches explicitly include
network connections as model parameters that must be optimized, they often rely
on explicit penalization or have network sparsity as a hyperparameter. This
work presents the Network Connectivity Gradients (NCG), a method to find
optimal connectivity patterns for RBMs based on the idea of network gradients:
computing the gradient of every possible connection, given a specific
connection pattern, and using the gradient to drive a continuous connection
strength parameter that in turn is used to determine the connection pattern.
Thus, learning RBM parameters and learning network connections is truly jointly
performed, albeit with different learning rates, and without changes to the
model's classic objective function. The method is applied to the MNIST and
other data sets showing that better RBM models are found for the benchmark
tasks of sample generation and input classification. Results also show that NCG
is robust to network initialization, both adding and removing network
connections while learning.
