  In video data, busy motion details from moving regions are conveyed within a
specific frequency bandwidth in the frequency domain. Meanwhile, the rest of
the frequencies of video data are encoded with quiet information with
substantial redundancy, which causes low processing efficiency in existing
video models that take as input raw RGB frames. In this paper, we consider
allocating intenser computation for the processing of the important busy
information and less computation for that of the quiet information. We design a
trainable Motion Band-Pass Module (MBPM) for separating busy information from
quiet information in raw video data. By embedding the MBPM into a two-pathway
CNN architecture, we define a Busy-Quiet Net (BQN). The efficiency of BQN is
determined by avoiding redundancy in the feature space processed by the two
pathways: one operating on Quiet features of low-resolution, while the other
processes Busy features. The proposed BQN outperforms many recent video
processing models on Something-Something V1, Kinetics400, UCF101 and HMDB51
datasets.
