  Advances in deep neural networks (DNNs) are transforming science and
technology. However, the increasing computational demands of the most powerful
DNNs limit deployment on low-power devices, such as smartphones and sensors --
and this trend is accelerated by the simultaneous move towards
Internet-of-Things (IoT) devices. Numerous efforts are underway to lower power
consumption, but a fundamental bottleneck remains due to energy consumption in
matrix algebra, even for analog approaches including neuromorphic, analog
memory and photonic meshes. Here we introduce and demonstrate a new approach
that sharply reduces energy required for matrix algebra by doing away with
weight memory access on edge devices, enabling orders of magnitude energy and
latency reduction. At the core of our approach is a new concept that
decentralizes the DNN for delocalized, optically accelerated matrix algebra on
edge devices. Using a silicon photonic smart transceiver, we demonstrate
experimentally that this scheme, termed Netcast, dramatically reduces energy
consumption. We demonstrate operation in a photon-starved environment with 40
aJ/multiply of optical energy for 98.8% accurate image recognition and <1
photon/multiply using single photon detectors. Furthermore, we show realistic
deployment of our system, classifying images with 3 THz of bandwidth over 86 km
of deployed optical fiber in a Boston-area fiber network. Our approach enables
computing on a new generation of edge devices with speeds comparable to modern
digital electronics and power consumption that is orders of magnitude lower.
