  Deep neural networks (DNNs) have been deployed in myriad machine learning
applications. However, advances in their accuracy are often achieved with
increasingly complex and deep network architectures. These large, deep models
are often unsuitable for real-world applications, due to their massive
computational cost, high memory bandwidth, and long latency. For example,
autonomous driving requires fast inference based on Internet-of-Things (IoT)
edge devices operating under run-time energy and memory storage constraints. In
such cases, compact DNNs can facilitate deployment due to their reduced energy
consumption, memory requirement, and inference latency. Long short-term
memories (LSTMs) are a type of recurrent neural network that have also found
widespread use in the context of sequential data modeling. They also face a
model size vs. accuracy trade-off. In this paper, we review major approaches
for automatically synthesizing compact, yet accurate, DNN/LSTM models suitable
for real-world applications. We also outline some challenges and future areas
of exploration.
