  In this paper, we consider domain-invariant deep learning by explicitly
modeling domain shifts with only a small amount of domain-specific parameters
in a Convolutional Neural Network (CNN). By exploiting the observation that a
convolutional filter can be well approximated as a linear combination of a
small set of dictionary atoms, we show for the first time, both empirically and
theoretically, that domain shifts can be effectively handled by decomposing a
convolutional layer into a domain-specific atom layer and a domain-shared
coefficient layer, while both remain convolutional. An input channel will now
first convolve spatially only with each respective domain-specific dictionary
atom to "absorb" domain variations, and then output channels are linearly
combined using common decomposition coefficients trained to promote shared
semantics across domains. We use toy examples, rigorous analysis, and
real-world examples with diverse datasets and architectures, to show the
proposed plug-in framework's effectiveness in cross and joint domain
performance and domain adaptation. With the proposed architecture, we need only
a small set of dictionary atoms to model each additional domain, which brings a
negligible amount of additional parameters, typically a few hundred.
