  Deep neural networks (DNNs) have inspired new studies in myriad edge
applications with robots, autonomous agents, and Internet-of-things (IoT)
devices. However, performing inference of DNNs in the edge is still a severe
challenge, mainly because of the contradiction between the intensive resource
requirements of DNNs and the tight resource availability in several edge
domains. Further, as communication is costly, taking advantage of other
available edge devices by using data- or model-parallelism methods is not an
effective solution. To benefit from available compute resources with low
communication overhead, we propose the first DNN parallelization method for
reducing the communication overhead in a distributed system. We propose a
low-communication parallelization (LCP) method in which models consist of
several almost-independent and narrow branches. LCP offers close-to-minimum
communication overhead with better distribution and parallelization
opportunities while significantly reducing memory footprint and computation
compared to data- and model-parallelism methods. We deploy LCP models on three
distributed systems: AWS instances, Raspberry Pis, and PYNQ boards. We also
evaluate the performance of LCP models on a customized hardware (tailored for
low latency) implemented on a small edge FPGA and as a 16mW 0.107mm2 ASIC @7nm
chip. LCP models achieve a maximum and average speedups of 56x and 7x, compared
to the originals, which could be improved by up to an average speedup of 33x by
incorporating common optimizations such as pruning and quantization.
