  The 3D occupancy estimation task has become an important challenge in the
area of vision-based autonomous driving recently. However, most existing
camera-based methods rely on costly 3D voxel labels or LiDAR scans for
training, limiting their practicality and scalability. Moreover, most methods
are tied to a predefined set of classes which they can detect. In this work we
present a novel approach for open vocabulary occupancy estimation called
LangOcc, that is trained only via camera images, and can detect arbitrary
semantics via vision-language alignment. In particular, we distill the
knowledge of the strong vision-language aligned encoder CLIP into a 3D
occupancy model via differentiable volume rendering. Our model estimates
vision-language aligned features in a 3D voxel grid using only images. It is
trained in a self-supervised manner by rendering our estimations back to 2D
space, where ground-truth features can be computed. This training mechanism
automatically supervises the scene geometry, allowing for a straight-forward
and powerful training method without any explicit geometry supervision. LangOcc
outperforms LiDAR-supervised competitors in open vocabulary occupancy by a
large margin, solely relying on vision-based training. We also achieve
state-of-the-art results in self-supervised semantic occupancy estimation on
the Occ3D-nuScenes dataset, despite not being limited to a specific set of
categories, thus demonstrating the effectiveness of our proposed
vision-language training.
