  Dynamic neural network is an emerging research topic in deep learning.
Compared to static models which have fixed computational graphs and parameters
at the inference stage, dynamic networks can adapt their structures or
parameters to different inputs, leading to notable advantages in terms of
accuracy, computational efficiency, adaptiveness, etc. In this survey, we
comprehensively review this rapidly developing area by dividing dynamic
networks into three main categories: 1) instance-wise dynamic models that
process each instance with data-dependent architectures or parameters; 2)
spatial-wise dynamic networks that conduct adaptive computation with respect to
different spatial locations of image data and 3) temporal-wise dynamic models
that perform adaptive inference along the temporal dimension for sequential
data such as videos and texts. The important research problems of dynamic
networks, e.g., architecture design, decision making scheme, optimization
technique and applications, are reviewed systematically. Finally, we discuss
the open problems in this field together with interesting future research
directions.
