  Knowledge Distillation has been established as a highly promising approach
for training compact and faster models by transferring knowledge from
heavyweight and powerful models. However, KD in its conventional version
constitutes an enduring, computationally and memory demanding process. In this
paper, Online Self-Acquired Knowledge Distillation (OSAKD) is proposed, aiming
to improve the performance of any deep neural model in an online manner. We
utilize k-nn non-parametric density estimation technique for estimating the
unknown probability distributions of the data samples in the output feature
space. This allows us for directly estimating the posterior class probabilities
of the data samples, and we use them as soft labels that encode explicit
information about the similarities of the data with the classes, negligibly
affecting the computational cost. The experimental evaluation on four datasets
validates the effectiveness of proposed method.
