  In this paper, we explore an alternate method for synthesizing neural network
architectures, inspired by the brain's stochastic synaptic pruning. During a
person's lifetime, numerous distinct neuronal architectures are responsible for
performing the same tasks. This indicates that biological neural networks are,
to some degree, architecture agnostic. However, artificial networks rely on
their fine-tuned weights and hand-crafted architectures for their remarkable
performance. This contrast begs the question: Can we build artificial
architecture agnostic neural networks? To ground this study we utilize sparse,
binary neural networks that parallel the brain's circuits. Within this sparse,
binary paradigm we sample many binary architectures to create families of
architecture agnostic neural networks not trained via backpropagation. These
high-performing network families share the same sparsity, distribution of
binary weights, and succeed in both static and dynamic tasks. In summation, we
create an architecture manifold search procedure to discover families or
architecture agnostic neural networks.
