  Deep learning techniques have been demonstrated to surpass preceding
cutting-edge machine learning techniques in recent years, with computer vision
being one of the most prominent examples. However, deep learning models suffer
from significant drawbacks when deployed in resource-constrained environments
due to their large model size and high complexity. Knowledge Distillation is
one of the prominent solutions to overcome this challenge. This review paper
examines the current state of research on knowledge distillation, a technique
for compressing complex models into smaller and simpler ones. The paper
provides an overview of the major principles and techniques associated with
knowledge distillation and reviews the applications of knowledge distillation
in the domain of computer vision. The review focuses on the benefits of
knowledge distillation, as well as the problems that must be overcome to
improve its effectiveness.
