  Most action recognition models today are highly parameterized, and evaluated
on datasets with appearance-wise distinct classes. It has also been shown that
2D Convolutional Neural Networks (CNNs) tend to be biased toward texture rather
than shape in still image recognition tasks, in contrast to humans. Taken
together, this raises suspicion that large video models partly learn spurious
spatial texture correlations rather than to track relevant shapes over time to
infer generalizable semantics from their movement. A natural way to avoid
parameter explosion when learning visual patterns over time is to make use of
recurrence. Biological vision consists of abundant recurrent circuitry, and is
superior to computer vision in terms of domain shift generalization. In this
article, we empirically study whether the choice of low-level temporal modeling
has consequences for texture bias and cross-domain robustness. In order to
enable a light-weight and systematic assessment of the ability to capture
temporal structure, not revealed from single frames, we provide the Temporal
Shape (TS) dataset, as well as modified domains of Diving48 allowing for the
investigation of spatial texture bias in video models. The combined results of
our experiments indicate that sound physical inductive bias such as recurrence
in temporal modeling may be advantageous when robustness to domain shift is
important for the task.
