  This paper strives for activity recognition under domain shift, for example
caused by change of scenery or camera viewpoint. The leading approaches reduce
the shift in activity appearance by adversarial training and self-supervised
learning. Different from these vision-focused works we leverage activity sounds
for domain adaptation as they have less variance across domains and can
reliably indicate which activities are not happening. We propose an
audio-adaptive encoder and associated learning methods that discriminatively
adjust the visual feature representation as well as addressing shifts in the
semantic distribution. To further eliminate domain-specific features and
include domain-invariant activity sounds for recognition, an audio-infused
recognizer is proposed, which effectively models the cross-modal interaction
across domains. We also introduce the new task of actor shift, with a
corresponding audio-visual dataset, to challenge our method with situations
where the activity appearance changes dramatically. Experiments on this
dataset, EPIC-Kitchens and CharadesEgo show the effectiveness of our approach.
