  Neural networks pose a privacy risk to training data due to their propensity
to memorise and leak information. Focusing on image classification, we show
that neural networks also unintentionally memorise unique features even when
they occur only once in training data. An example of a unique feature is a
person's name that is accidentally present on a training image. Assuming access
to the inputs and outputs of a trained model, the domain of the training data,
and knowledge of unique features, we develop a score estimating the model's
sensitivity to a unique feature by comparing the KL divergences of the model's
output distributions given modified out-of-distribution images. Our results
suggest that unique features are memorised by multi-layer perceptrons and
convolutional neural networks trained on benchmark datasets, such as MNIST,
Fashion-MNIST and CIFAR-10. We find that strategies to prevent overfitting
(e.g.\ early stopping, regularisation, batch normalisation) do not prevent
memorisation of unique features. These results imply that neural networks pose
a privacy risk to rarely occurring private information. These risks can be more
pronounced in healthcare applications if patient information is present in the
training data.
