  Recent years have witnessed the great success of convolutional neural network
(CNN) based models in the field of computer vision. CNN is able to learn
hierarchically abstracted features from images in an end-to-end training
manner. However, most of the existing CNN models only learn features through a
feedforward structure and no feedback information from top to bottom layers is
exploited to enable the networks to refine themselves. In this paper, we
propose a "Learning with Rethinking" algorithm. By adding a feedback layer and
producing the emphasis vector, the model is able to recurrently boost the
performance based on previous prediction. Particularly, it can be employed to
boost any pre-trained models. This algorithm is tested on four object
classification benchmark datasets: CIFAR-100, CIFAR-10, MNIST-background-image
and ILSVRC-2012 dataset. These results have demonstrated the advantage of
training CNN models with the proposed feedback mechanism.
