  One significant factor we expect the video representation learning to
capture, especially in contrast with the image representation learning, is the
object motion. However, we found that in the current mainstream video datasets,
some action categories are highly related with the scene where the action
happens, making the model tend to degrade to a solution where only the scene
information is encoded. For example, a trained model may predict a video as
playing football simply because it sees the field, neglecting that the subject
is dancing as a cheerleader on the field. This is against our original
intention towards the video representation learning and may bring scene bias on
different dataset that can not be ignored. In order to tackle this problem, we
propose to decouple the scene and the motion (DSM) with two simple operations,
so that the model attention towards the motion information is better paid.
Specifically, we construct a positive clip and a negative clip for each video.
Compared to the original video, the positive/negative is
motion-untouched/broken but scene-broken/untouched by Spatial Local Disturbance
and Temporal Local Disturbance. Our objective is to pull the positive closer
while pushing the negative farther to the original clip in the latent space. In
this way, the impact of the scene is weakened while the temporal sensitivity of
the network is further enhanced. We conduct experiments on two tasks with
various backbones and different pre-training datasets, and find that our method
surpass the SOTA methods with a remarkable 8.1% and 8.8% improvement towards
action recognition task on the UCF101 and HMDB51 datasets respectively using
the same backbone.
