  Optimizing resource utilization in target platforms is key to achieving high
performance during DNN inference. While optimizations have been proposed for
inference latency, memory footprint, and energy consumption, prior
hardware-aware neural architecture search (NAS) methods have omitted resource
utilization, preventing DNNs to take full advantage of the target inference
platforms. Modeling resource utilization efficiently and accurately is
challenging, especially for widely-used array-based inference accelerators such
as Google TPU. In this work, we propose a novel hardware-aware NAS framework
that does not only optimize for task accuracy and inference latency, but also
for resource utilization. We also propose and validate a new computational
model for resource utilization in inference accelerators. By using the proposed
NAS framework and the proposed resource utilization model, we achieve 2.8 - 4x
speedup for DNN inference compared to prior hardware-aware NAS methods while
attaining similar or improved accuracy in image classification on CIFAR-10 and
Imagenet-100 datasets.
