  Recent work has shown that using unlabeled data in semi-supervised learning
is not always beneficial and can even hurt generalization, especially when
there is a class mismatch between the unlabeled and labeled examples. We
investigate this phenomenon for image classification on the CIFAR-10 and the
ImageNet datasets, and with many other forms of domain shifts applied (e.g.
salt-and-pepper noise). Our main contribution is Split Batch Normalization
(Split-BN), a technique to improve SSL when the additional unlabeled data comes
from a shifted distribution. We achieve it by using separate batch
normalization statistics for unlabeled examples. Due to its simplicity, we
recommend it as a standard practice. Finally, we analyse how domain shift
affects the SSL training process. In particular, we find that during training
the statistics of hidden activations in late layers become markedly different
between the unlabeled and the labeled examples.
