  Effective training of deep neural networks can be challenging, and there
remain many open questions on how to best learn these models. Recently
developed methods to improve neural network training examine teaching:
providing learned information during the training process to improve downstream
model performance. In this paper, we take steps towards extending the scope of
teaching. We propose a flexible teaching framework using commentaries, learned
meta-information helpful for training on a particular task. We present
gradient-based methods to learn commentaries, leveraging recent work on
implicit differentiation for scalability. We explore diverse applications of
commentaries, from weighting training examples, to parameterising
label-dependent data augmentation policies, to representing attention masks
that highlight salient image regions. We find that commentaries can improve
training speed and/or performance, and provide insights about the dataset and
training process. We also observe that commentaries generalise: they can be
reused when training new models to obtain performance benefits, suggesting a
use-case where commentaries are stored with a dataset and leveraged in future
for improved model training.
