  Domain adaptation techniques, which focus on adapting models between
distributionally different domains, are rarely explored in the video
recognition area due to the significant spatial and temporal shifts across the
source (i.e. training) and target (i.e. test) domains. As such, recent works on
visual domain adaptation which leverage adversarial learning to unify the
source and target video representations and strengthen the feature
transferability are not highly effective on the videos. To overcome this
limitation, in this paper, we learn a domain-agnostic video classifier instead
of learning domain-invariant representations, and propose an Adversarial
Bipartite Graph (ABG) learning framework which directly models the
source-target interactions with a network topology of the bipartite graph.
Specifically, the source and target frames are sampled as heterogeneous
vertexes while the edges connecting two types of nodes measure the affinity
among them. Through message-passing, each vertex aggregates the features from
its heterogeneous neighbors, forcing the features coming from the same class to
be mixed evenly. Explicitly exposing the video classifier to such cross-domain
representations at the training and test stages makes our model less biased to
the labeled source data, which in-turn results in achieving a better
generalization on the target domain. To further enhance the model capacity and
testify the robustness of the proposed architecture on difficult transfer
tasks, we extend our model to work in a semi-supervised setting using an
additional video-level bipartite graph. Extensive experiments conducted on four
benchmarks evidence the effectiveness of the proposed approach over the SOTA
methods on the task of video recognition.
