  This paper gives an overview of our ongoing work on the design space
exploration of efficient deep neural networks (DNNs). Specifically, we cover
two aspects: (1) static architecture design efficiency and (2) dynamic model
execution efficiency. For static architecture design, different from existing
end-to-end hardware modeling assumptions, we conduct full-stack profiling at
the GPU core level to identify better accuracy-latency trade-offs for DNN
designs. For dynamic model execution, different from prior work that tackles
model redundancy at the DNN-channels level, we explore a new dimension of DNN
feature map redundancy to be dynamically traversed at runtime. Last, we
highlight several open questions that are poised to draw research attention in
the next few years.
