  The number of parameters in deep neural networks (DNNs) is scaling at about
5$\times$ the rate of Moore's Law. To sustain this growth, photonic computing
is a promising avenue, as it enables higher throughput in dominant general
matrix-matrix multiplication (GEMM) operations in DNNs than their electrical
counterpart. However, purely photonic systems face several challenges including
lack of photonic memory and accumulation of noise. In this paper, we present an
electro-photonic accelerator, ADEPT, which leverages a photonic computing unit
for performing GEMM operations, a vectorized digital electronic ASIC for
performing non-GEMM operations, and SRAM arrays for storing DNN parameters and
activations. In contrast to prior works in photonic DNN accelerators, we adopt
a system-level perspective and show that the gains while large are tempered
relative to prior expectations. Our goal is to encourage architects to explore
photonic technology in a more pragmatic way considering the system as a whole
to understand its general applicability in accelerating today's DNNs. Our
evaluation shows that ADEPT can provide, on average, 5.73$\times$ higher
throughput per Watt compared to the traditional systolic arrays (SAs) in a
full-system, and at least 6.8$\times$ and $2.5\times$ better throughput per
Watt, compared to state-of-the-art electronic and photonic accelerators,
respectively.
