  Most existing studies on unsupervised domain adaptation (UDA) assume that
each domain's training samples come with domain labels (e.g., painting, photo).
Samples from each domain are assumed to follow the same distribution and the
domain labels are exploited to learn domain-invariant features via feature
alignment. However, such an assumption often does not hold true -- there often
exist numerous finer-grained domains (e.g., dozens of modern painting styles
have been developed, each differing dramatically from those of the classic
styles). Therefore, forcing feature distribution alignment across each
artificially-defined and coarse-grained domain can be ineffective. In this
paper, we address both single-source and multi-source UDA from a completely
different perspective, which is to view each instance as a fine domain. Feature
alignment across domains is thus redundant. Instead, we propose to perform
dynamic instance domain adaptation (DIDA). Concretely, a dynamic neural network
with adaptive convolutional kernels is developed to generate instance-adaptive
residuals to adapt domain-agnostic deep features to each individual instance.
This enables a shared classifier to be applied to both source and target domain
data without relying on any domain annotation. Further, instead of imposing
intricate feature alignment losses, we adopt a simple semi-supervised learning
paradigm using only a cross-entropy loss for both labeled source and pseudo
labeled target data. Our model, dubbed DIDA-Net, achieves state-of-the-art
performance on several commonly used single-source and multi-source UDA
datasets including Digits, Office-Home, DomainNet, Digit-Five, and PACS.
