  Batch Normalization's (BN) unique property of depending on other samples in a
batch is known to cause problems in several tasks, including sequence modeling.
Yet, BN-related issues are hardly studied for long video understanding, despite
the ubiquitous use of BN in CNNs (Convolutional Neural Networks) for feature
extraction. Especially in surgical workflow analysis, where the lack of
pretrained feature extractors has led to complex, multi-stage training
pipelines, limited awareness of BN issues may have hidden the benefits of
training CNNs and temporal models end to end. In this paper, we analyze
pitfalls of BN in video learning, including issues specific to online tasks
such as a 'cheating' effect in anticipation. We observe that BN's properties
create major obstacles for end-to-end learning. However, using BN-free
backbones, even simple CNN-LSTMs beat the state of the art
{\color{\colorrevtwo}on three surgical workflow benchmarks} by utilizing
adequate end-to-end training strategies which maximize temporal context. We
conclude that awareness of BN's pitfalls is crucial for effective end-to-end
learning in surgical tasks. By reproducing results on natural-video datasets,
we hope our insights will benefit other areas of video learning as well. Code
is available at: \url{https://gitlab.com/nct_tso_public/pitfalls_bn}
