  Although action recognition systems can achieve top performance when
evaluated on in-distribution test points, they are vulnerable to unanticipated
distribution shifts in test data. However, test-time adaptation of video action
recognition models against common distribution shifts has so far not been
demonstrated. We propose to address this problem with an approach tailored to
spatio-temporal models that is capable of adaptation on a single video sample
at a step. It consists in a feature distribution alignment technique that
aligns online estimates of test set statistics towards the training statistics.
We further enforce prediction consistency over temporally augmented views of
the same test video sample. Evaluations on three benchmark action recognition
datasets show that our proposed technique is architecture-agnostic and able to
significantly boost the performance on both, the state of the art convolutional
architecture TANet and the Video Swin Transformer. Our proposed method
demonstrates a substantial performance gain over existing test-time adaptation
approaches in both evaluations of a single distribution shift and the
challenging case of random distribution shifts. Code will be available at
\url{https://github.com/wlin-at/ViTTA}.
