  Customizing Convolution Neural Networks (CNN) for production use has been a
challenging task for DL practitioners. This paper intends to expedite the model
customization with a model hub that contains the optimized models tiered by
their inference latency using Neural Architecture Search (NAS). To achieve this
goal, we build a distributed NAS system to search on a novel search space that
consists of prominent factors to impact latency and accuracy. Since we target
GPU, we name the NAS optimized models as GPUNet, which establishes a new SOTA
Pareto frontier in inference latency and accuracy. Within 1$ms$, GPUNet is 2x
faster than EfficientNet-X and FBNetV3 with even better accuracy. We also
validate GPUNet on detection tasks, and GPUNet consistently outperforms
EfficientNet-X and FBNetV3 on COCO detection tasks in both latency and
accuracy. All of these data validate that our NAS system is effective and
generic to handle different design tasks. With this NAS system, we expand
GPUNet to cover a wide range of latency targets such that DL practitioners can
deploy our models directly in different scenarios.
