  With the rise in edge-computing devices, there has been an increasing demand
to deploy energy and resource-efficient models. A large body of research has
been devoted to developing methods that can reduce the size of the model
considerably without affecting the standard metrics such as top-1 accuracy.
However, these pruning approaches tend to result in a significant mismatch in
other metrics such as fairness across classes and explainability. To combat
such misalignment, we propose a novel multi-part loss function inspired by the
knowledge-distillation literature. Through extensive experiments, we
demonstrate the effectiveness of our approach across different compression
algorithms, architectures, tasks as well as datasets. In particular, we obtain
up to $4.1\times$ reduction in the number of prediction mismatches between the
compressed and reference models, and up to $5.7\times$ in cases where the
reference model makes the correct prediction; all while making no changes to
the compression algorithm, and minor modifications to the loss function.
Furthermore, we demonstrate how inducing simple alignment between the
predictions of the models naturally improves the alignment on other metrics
including fairness and attributions. Our framework can thus serve as a simple
plug-and-play component for compression algorithms in the future.
