  Reducing inconsistencies in the behavior of different versions of an AI
system can be as important in practice as reducing its overall error. In image
classification, sample-wise inconsistencies appear as "negative flips": A new
model incorrectly predicts the output for a test sample that was correctly
classified by the old (reference) model. Positive-congruent (PC) training aims
at reducing error rate while at the same time reducing negative flips, thus
maximizing congruency with the reference model only on positive predictions,
unlike model distillation. We propose a simple approach for PC training, Focal
Distillation, which enforces congruence with the reference model by giving more
weights to samples that were correctly classified. We also found that, if the
reference model itself can be chosen as an ensemble of multiple deep neural
networks, negative flips can be further reduced without affecting the new
model's accuracy.
