  Neural Architecture Search (NAS) has been widely adopted to design accurate
and efficient image classification models. However, applying NAS to a new
computer vision task still requires a huge amount of effort. This is because 1)
previous NAS research has been over-prioritized on image classification while
largely ignoring other tasks; 2) many NAS works focus on optimizing
task-specific components that cannot be favorably transferred to other tasks;
and 3) existing NAS methods are typically designed to be "proxyless" and
require significant effort to be integrated with each new task's training
pipelines. To tackle these challenges, we propose FBNetV5, a NAS framework that
can search for neural architectures for a variety of vision tasks with much
reduced computational cost and human effort. Specifically, we design 1) a
search space that is simple yet inclusive and transferable; 2) a multitask
search process that is disentangled with target tasks' training pipeline; and
3) an algorithm to simultaneously search for architectures for multiple tasks
with a computational cost agnostic to the number of tasks. We evaluate the
proposed FBNetV5 targeting three fundamental vision tasks -- image
classification, object detection, and semantic segmentation. Models searched by
FBNetV5 in a single run of search have outperformed the previous
stateof-the-art in all the three tasks: image classification (e.g., +1.3%
ImageNet top-1 accuracy under the same FLOPs as compared to FBNetV3), semantic
segmentation (e.g., +1.8% higher ADE20K val. mIoU than SegFormer with 3.6x
fewer FLOPs), and object detection (e.g., +1.1% COCO val. mAP with 1.2x fewer
FLOPs as compared to YOLOX).
