  The performance of a classifier trained on data coming from a specific domain
typically degrades when applied to a related but different one. While
annotating many samples from the new domain would address this issue, it is
often too expensive or impractical. Domain Adaptation has therefore emerged as
a solution to this problem; It leverages annotated data from a source domain,
in which it is abundant, to train a classifier to operate in a target domain,
in which it is either sparse or even lacking altogether. In this context, the
recent trend consists of learning deep architectures whose weights are shared
for both domains, which essentially amounts to learning domain invariant
features.
  Here, we show that it is more effective to explicitly model the shift from
one domain to the other. To this end, we introduce a two-stream architecture,
where one operates in the source domain and the other in the target domain. In
contrast to other approaches, the weights in corresponding layers are related
but not shared. We demonstrate that this both yields higher accuracy than
state-of-the-art methods on several object recognition and detection tasks and
consistently outperforms networks with shared weights in both supervised and
unsupervised settings.
