  BatchNorm is a critical building block in modern convolutional neural
networks. Its unique property of operating on "batches" instead of individual
samples introduces significantly different behaviors from most other operations
in deep learning. As a result, it leads to many hidden caveats that can
negatively impact model's performance in subtle ways. This paper thoroughly
reviews such problems in visual recognition tasks, and shows that a key to
address them is to rethink different choices in the concept of "batch" in
BatchNorm. By presenting these caveats and their mitigations, we hope this
review can help researchers use BatchNorm more effectively.
