  There are various optimization techniques in the realm of 3D, including point
cloud-based approaches that use mesh, texture, and voxels which optimize how
you store, and how do calculate in 3D. These techniques employ methods such as
feed-forward networks, 3D convolutions, graph neural networks, transformers,
and sparse tensors. However, the field of 3D is one of the most computationally
expensive fields, and these methods have yet to achieve their full potential
due to their large capacity, complexity, and computation limits. This paper
proposes the application of knowledge distillation techniques, especially for
sparse tensors in 3D deep learning, to reduce model sizes while maintaining
performance. We analyze and purpose different loss functions, including
standard methods and combinations of various losses, to simulate the
performance of state-of-the-art models of different Sparse Convolutional NNs.
Our experiments are done on the standard ScanNet V2 dataset, and we achieved
around 2.6\% mIoU difference with a 4 times smaller model and around 8\% with a
16 times smaller model on the latest state-of-the-art spacio-temporal convents
based models.
