  We consider efficiency in the implementation of deep neural networks.
Hardware accelerators are gaining interest as machine learning becomes one of
the drivers of high-performance computing. In these accelerators, the directed
graph describing a neural network can be implemented as a directed graph
describing a Boolean circuit. We make this observation precise, leading
naturally to an understanding of practical neural networks as discrete
functions, and show that so-called binarised neural networks are functionally
complete. In general, our results suggest that it is valuable to consider
Boolean circuits as neural networks, leading to the question of which circuit
topologies are promising. We argue that continuity is central to generalisation
in learning, explore the interaction between data coding, network topology, and
node functionality for continuity, and pose some open questions for future
research. As a first step to bridging the gap between continuous and Boolean
views of neural network accelerators, we present some recent results from our
work on LUTNet, a novel Field-Programmable Gate Array inference approach.
Finally, we conclude with additional possible fruitful avenues for research
bridging the continuous and discrete views of neural networks.
