  Batch normalization (BN) uniformly shifts and scales the activations based on
the statistics of a batch of images. However, the intensity distribution of the
background pixels often dominates the BN statistics because the background
accounts for a large proportion of the entire image. This paper focuses on
enhancing BN with the intensity distribution of foreground pixels, the one that
really matters for image segmentation. We propose a new normalization strategy,
named categorical normalization (CateNorm), to normalize the activations
according to categorical statistics. The categorical statistics are obtained by
dynamically modulating specific regions in an image that belong to the
foreground. CateNorm demonstrates both precise and robust segmentation results
across five public datasets obtained from different domains, covering complex
and variable data distributions. It is attributable to the ability of CateNorm
to capture domain-invariant information from multiple domains (institutions) of
medical data. Code is available at https://github.com/lambert-x/CateNorm.
