  This research aims to explore the possibility of designing a neural network
architecture that allows for small networks to adopt the properties of huge
networks, which have shown success in self-supervised learning (SSL), for all
the downstream tasks like image classification, segmentation, etc. Previous
studies have shown that using convolutional neural networks (ConvNets) can
provide inherent inductive bias, which is crucial for learning representations
in deep learning models. To reduce the number of parameters, attention
mechanisms are utilized through the usage of MobileViT blocks, resulting in a
model with less than 5 million parameters. The model is trained using
self-distillation with momentum encoder and a student-teacher architecture is
also employed, where the teacher weights use vision transformers (ViTs) from
recent SOTA SSL models. The model is trained on the ImageNet1k dataset. This
research provides an approach for designing smaller, more efficient neural
network architectures that can perform SSL tasks comparable to heavy models
