  Biological intelligence systems of animals perceive the world by integrating
information in different modalities and processing simultaneously for various
tasks. In contrast, current machine learning research follows a task-specific
paradigm, leading to inefficient collaboration between tasks and high marginal
costs of developing perception models for new tasks. In this paper, we present
a generic perception architecture named Uni-Perceiver, which processes a
variety of modalities and tasks with unified modeling and shared parameters.
Specifically, Uni-Perceiver encodes different task inputs and targets from
arbitrary modalities into a unified representation space with a
modality-agnostic Transformer encoder and lightweight modality-specific
tokenizers. Different perception tasks are modeled as the same formulation,
that is, finding the maximum likelihood target for each input through the
similarity of their representations. The model is pre-trained on several
uni-modal and multi-modal tasks, and evaluated on a variety of downstream
tasks, including novel tasks that did not appear in the pre-training stage.
Results show that our pre-trained model without any tuning can achieve
reasonable performance even on novel tasks. The performance can be improved to
a level close to state-of-the-art methods by conducting prompt tuning on 1% of
downstream task data. Full-data fine-tuning further delivers results on par
with or better than state-of-the-art results. Code shall be released.
