  There is a growing need to deploy machine learning for different tasks on a
wide array of new hardware platforms. Such deployment scenarios require
tackling multiple challenges, including identifying a model architecture that
can achieve a suitable predictive accuracy (architecture search), and finding
an efficient implementation of the model to satisfy underlying
hardware-specific systems constraints such as latency (system optimization
search). Existing works treat architecture search and system optimization
search as separate problems and solve them sequentially. In this paper, we
instead propose to solve these problems jointly, and introduce a simple but
effective baseline method called SONAR that interleaves these two search
problems. SONAR aims to efficiently optimize for predictive accuracy and
inference latency by applying early stopping to both search processes. Our
experiments on multiple different hardware back-ends show that SONAR identifies
nearly optimal architectures 30 times faster than a brute force approach.
