  To build an artificial neural network like the biological intelligence
system, recent works have unified numerous tasks into a generalist model, which
can process various tasks with shared parameters and do not have any
task-specific modules. While generalist models achieve promising results on
various benchmarks, they have performance degradation on some tasks compared
with task-specialized models. In this work, we find that interference among
different tasks and modalities is the main factor to this phenomenon. To
mitigate such interference, we introduce the Conditional Mixture-of-Experts
(Conditional MoEs) to generalist models. Routing strategies under different
levels of conditions are proposed to take both the training/inference cost and
generalization ability into account. By incorporating the proposed Conditional
MoEs, the recently proposed generalist model Uni-Perceiver can effectively
mitigate the interference across tasks and modalities, and achieves
state-of-the-art results on a series of downstream tasks via prompt tuning on
1% of downstream data. Moreover, the introduction of Conditional MoEs still
holds the generalization ability of generalist models to conduct zero-shot
inference on new tasks, e.g., video-text retrieval and video caption. Code and
pre-trained generalist models shall be released.
