  In this paper, we reveal the two sides of data augmentation: enhancements in
closed-set recognition correlate with a significant decrease in open-set
recognition. Through empirical investigation, we find that multi-sample-based
augmentations would contribute to reducing feature discrimination, thereby
diminishing the open-set criteria. Although knowledge distillation could impair
the feature via imitation, the mixed feature with ambiguous semantics hinders
the distillation. To this end, we propose an asymmetric distillation framework
by feeding teacher model extra raw data to enlarge the benefit of teacher.
Moreover, a joint mutual information loss and a selective relabel strategy are
utilized to alleviate the influence of hard mixed samples. Our method
successfully mitigates the decline in open-set and outperforms SOTAs by 2%~3%
AUROC on the Tiny-ImageNet dataset and experiments on large-scale dataset
ImageNet-21K demonstrate the generalization of our method.
