  Batch Normalization (BN), a widely-used technique in neural networks,
enhances generalization and expedites training by normalizing each mini-batch
to the same mean and variance. However, its effectiveness diminishes when
confronted with diverse data distributions. To address this challenge, we
propose Supervised Batch Normalization (SBN), a pioneering approach. We expand
normalization beyond traditional single mean and variance parameters, enabling
the identification of data modes prior to training. This ensures effective
normalization for samples sharing common features. We define contexts as modes,
categorizing data with similar characteristics. These contexts are explicitly
defined, such as domains in domain adaptation or modalities in multimodal
systems, or implicitly defined through clustering algorithms based on data
similarity. We illustrate the superiority of our approach over BN and other
commonly employed normalization techniques through various experiments on both
single and multi-task datasets. Integrating SBN with Vision Transformer results
in a remarkable \textit{15.13}\% accuracy enhancement on CIFAR-100.
Additionally, in domain adaptation scenarios, employing AdaMatch demonstrates
an impressive \textit{22.25}\% accuracy improvement on MNIST and SVHN compared
to BN.
