  This article sets forth a review of knowledge distillation techniques with a
focus on their applicability to retail banking contexts. Predictive machine
learning algorithms used in banking environments, especially in risk and
control functions, are generally subject to regulatory and technical
constraints limiting their complexity. Knowledge distillation gives the
opportunity to improve the performances of simple models without burdening
their application, using the results of other - generally more complex and
better-performing - models. Parsing recent advances in this field, we highlight
three main approaches: Soft Targets, Sample Selection and Data Augmentation. We
assess the relevance of a subset of such techniques by applying them to open
source datasets, before putting them to the test on the use cases of BPCE, a
major French institution in the retail banking sector. As such, we demonstrate
the potential of knowledge distillation to improve the performance of these
models without altering their form and simplicity.
