  With the growing workload of inference tasks on mobile devices,
state-of-the-art neural architectures (NAs) are typically designed through
Neural Architecture Search (NAS) to identify NAs with good tradeoffs between
accuracy and efficiency (e.g., latency). Since measuring the latency of a huge
set of candidate architectures during NAS is not scalable, approaches are
needed for predicting end-to-end inference latency on mobile devices. Such
predictions are challenging due to hardware heterogeneity, optimizations
applied by ML frameworks, and the diversity of neural architectures. Motivated
by these challenges, in this paper, we first quantitatively assess
characteristics of neural architectures and mobile devices that have
significant effects on inference latency. Based on this assessment, we propose
a latency prediction framework which addresses these challenges by developing
operation-wise latency predictors, under a variety of settings and a number of
hardware devices, with multi-core CPUs and GPUs, achieving high accuracy in
end-to-end latency prediction, as shown by our comprehensive evaluations. To
illustrate that our approach does not require expensive data collection, we
also show that accurate predictions can be achieved on real-world NAs using
only small amounts of profiling data.
