  Deep neural networks have a clear degradation when applying to the unseen
environment due to the covariate shift. Conventional approaches like domain
adaptation requires the pre-collected target data for iterative training, which
is impractical in real-world applications. In this paper, we propose to adapt
the deep models to the novel environment during inference. An previous solution
is test time normalization, which substitutes the source statistics in BN
layers with the target batch statistics. However, we show that test time
normalization may potentially deteriorate the discriminative structures due to
the mismatch between target batch statistics and source parameters. To this
end, we present a general formulation $\alpha$-BN to calibrate the batch
statistics by mixing up the source and target statistics for both alleviating
the domain shift and preserving the discriminative structures. Based on
$\alpha$-BN, we further present a novel loss function to form a unified test
time adaptation framework Core, which performs the pairwise class correlation
online optimization. Extensive experiments show that our approaches achieve the
state-of-the-art performance on total twelve datasets from three topics,
including model robustness to corruptions, domain generalization on image
classification and semantic segmentation. Particularly, our $\alpha$-BN
improves 28.4\% to 43.9\% on GTA5 $\rightarrow$ Cityscapes without any
training, even outperforms the latest source-free domain adaptation method.
