  Video data is with complex temporal dynamics due to various factors such as
camera motion, speed variation, and different activities. To effectively
capture this diverse motion pattern, this paper presents a new temporal
adaptive module ({\bf TAM}) to generate video-specific temporal kernels based
on its own feature map. TAM proposes a unique two-level adaptive modeling
scheme by decoupling the dynamic kernel into a location sensitive importance
map and a location invariant aggregation weight. The importance map is learned
in a local temporal window to capture short-term information, while the
aggregation weight is generated from a global view with a focus on long-term
structure. TAM is a modular block and could be integrated into 2D CNNs to yield
a powerful video architecture (TANet) with a very small extra computational
cost. The extensive experiments on Kinetics-400 and Something-Something
datasets demonstrate that our TAM outperforms other temporal modeling methods
consistently, and achieves the state-of-the-art performance under the similar
complexity. The code is available at \url{
https://github.com/liu-zhy/temporal-adaptive-module}.
