  This work explores how to design a single neural network capable of adapting
to multiple heterogeneous vision tasks, such as image segmentation, 3D
detection, and video recognition. This goal is challenging because both network
architecture search (NAS) spaces and methods in different tasks are
inconsistent. We solve this challenge from both sides. We first introduce a
unified design space for multiple tasks and build a multitask NAS benchmark
(NAS-Bench-MR) on many widely used datasets, including ImageNet, Cityscapes,
KITTI, and HMDB51. We further propose Network Coding Propagation (NCP), which
back-propagates gradients of neural predictors to directly update architecture
codes along the desired gradient directions to solve various tasks. In this
way, optimal architecture configurations can be found by NCP in our large
search space in seconds.
  Unlike prior arts of NAS that typically focus on a single task, NCP has
several unique benefits. (1) NCP transforms architecture optimization from
data-driven to architecture-driven, enabling joint search an architecture among
multitasks with different data distributions. (2) NCP learns from network codes
but not original data, enabling it to update the architecture efficiently
across datasets. (3) In addition to our NAS-Bench-MR, NCP performs well on
other NAS benchmarks, such as NAS-Bench-201. (4) Thorough studies of NCP on
inter-, cross-, and intra-tasks highlight the importance of cross-task neural
architecture design, i.e., multitask neural architectures and architecture
transferring between different tasks. Code is available at
https://github.com/dingmyu/NCP.
