  Attention plays a critical role in human visual experience. Furthermore, it
has recently been demonstrated that attention can also play an important role
in the context of applying artificial neural networks to a variety of tasks
from fields such as computer vision and NLP. In this work we show that, by
properly defining attention for convolutional neural networks, we can actually
use this type of information in order to significantly improve the performance
of a student CNN network by forcing it to mimic the attention maps of a
powerful teacher network. To that end, we propose several novel methods of
transferring attention, showing consistent improvement across a variety of
datasets and convolutional neural network architectures. Code and models for
our experiments are available at
https://github.com/szagoruyko/attention-transfer
