  This paper discusses four facets of the Knowledge Distillation (KD) process
for Convolutional Neural Networks (CNNs) and Vision Transformer (ViT)
architectures, particularly when executed on edge devices with constrained
processing capabilities. First, we conduct a comparative analysis of the KD
process between CNNs and ViT architectures, aiming to elucidate the feasibility
and efficacy of employing different architectural configurations for the
teacher and student, while assessing their performance and efficiency. Second,
we explore the impact of varying the size of the student model on accuracy and
inference speed, while maintaining a constant KD duration. Third, we examine
the effects of employing higher resolution images on the accuracy, memory
footprint and computational workload. Last, we examine the performance
improvements obtained by fine-tuning the student model after KD to specific
downstream tasks. Through empirical evaluations and analyses, this research
provides AI practitioners with insights into optimal strategies for maximizing
the effectiveness of the KD process on edge devices.
