  Early exiting has become a promising approach to improving the inference
efficiency of deep networks. By structuring models with multiple classifiers
(exits), predictions for ``easy'' samples can be generated at earlier exits,
negating the need for executing deeper layers. Current multi-exit networks
typically implement linear classifiers at intermediate layers, compelling
low-level features to encapsulate high-level semantics. This sub-optimal design
invariably undermines the performance of later exits. In this paper, we propose
Dynamic Perceiver (Dyn-Perceiver) to decouple the feature extraction procedure
and the early classification task with a novel dual-branch architecture. A
feature branch serves to extract image features, while a classification branch
processes a latent code assigned for classification tasks. Bi-directional
cross-attention layers are established to progressively fuse the information of
both branches. Early exits are placed exclusively within the classification
branch, thus eliminating the need for linear separability in low-level
features. Dyn-Perceiver constitutes a versatile and adaptable framework that
can be built upon various architectures. Experiments on image classification,
action recognition, and object detection demonstrate that our method
significantly improves the inference efficiency of different backbones,
outperforming numerous competitive approaches across a broad range of
computational budgets. Evaluation on both CPU and GPU platforms substantiate
the superior practical efficiency of Dyn-Perceiver. Code is available at
https://www.github.com/LeapLabTHU/Dynamic_Perceiver.
