  Unsupervised learning and supervised learning are key research topics in deep
learning. However, as high-capacity supervised neural networks trained with a
large amount of labels have achieved remarkable success in many computer vision
tasks, the availability of large-scale labeled images reduced the significance
of unsupervised learning. Inspired by the recent trend toward revisiting the
importance of unsupervised learning, we investigate joint supervised and
unsupervised learning in a large-scale setting by augmenting existing neural
networks with decoding pathways for reconstruction. First, we demonstrate that
the intermediate activations of pretrained large-scale classification networks
preserve almost all the information of input images except a portion of local
spatial details. Then, by end-to-end training of the entire augmented
architecture with the reconstructive objective, we show improvement of the
network performance for supervised tasks. We evaluate several variants of
autoencoders, including the recently proposed "what-where" autoencoder that
uses the encoder pooling switches, to study the importance of the architecture
design. Taking the 16-layer VGGNet trained under the ImageNet ILSVRC 2012
protocol as a strong baseline for image classification, our methods improve the
validation-set accuracy by a noticeable margin.
