  Compressing giant neural networks has gained much attention for their
extensive applications on edge devices such as cellphones. During the
compressing process, one of the most important procedures is to retrain the
pre-trained models using the original training dataset. However, due to the
consideration of security, privacy or commercial profits, in practice, only a
fraction of sample training data are made available, which makes the retraining
infeasible. To solve this issue, this paper proposes to resort to unlabeled
data in hand that can be cheaper to acquire. Specifically, we exploit the
unlabeled data to mimic the classification characteristics of giant networks,
so that the original capacity can be preserved nicely. Nevertheless, there
exists a dataset bias between the labeled and unlabeled data, which may disturb
the training and degrade the performance. We thus fix this bias by an
adversarial loss to make an alignment on the distributions of their low-level
feature representations. We further provide theoretical discussions about how
the unlabeled data help compressed networks to generalize better. Experimental
results demonstrate that the unlabeled data can significantly improve the
performance of the compressed networks.
