  Recently, a number of iterative learning methods have been introduced to
improve generalization. These typically rely on training for longer periods of
time in exchange for improved generalization. LLF (later-layer-forgetting) is a
state-of-the-art method in this category. It strengthens learning in early
layers by periodically re-initializing the last few layers of the network. Our
principal innovation in this work is to use Simulated annealing in EArly Layers
(SEAL) of the network in place of re-initialization of later layers.
Essentially, later layers go through the normal gradient descent process, while
the early layers go through short stints of gradient ascent followed by
gradient descent. Extensive experiments on the popular Tiny-ImageNet dataset
benchmark and a series of transfer learning and few-shot learning tasks show
that we outperform LLF by a significant margin. We further show that, compared
to normal training, LLF features, although improving on the target task,
degrade the transfer learning performance across all datasets we explored. In
comparison, our method outperforms LLF across the same target datasets by a
large margin. We also show that the prediction depth of our method is
significantly lower than that of LLF and normal training, indicating on average
better prediction performance.
