  3D reconstruction has been widely used in autonomous navigation fields of
mobile robotics. However, the former research can only provide the basic
geometry structure without the capability of open-world scene understanding,
limiting advanced tasks like human interaction and visual navigation. Moreover,
traditional 3D scene understanding approaches rely on expensive labeled 3D
datasets to train a model for a single task with supervision. Thus, geometric
reconstruction with zero-shot scene understanding i.e. Open vocabulary 3D
Understanding and Reconstruction, is crucial for the future development of
mobile robots. In this paper, we propose OpenOcc, a novel framework unifying
the 3D scene reconstruction and open vocabulary understanding with neural
radiance fields. We model the geometric structure of the scene with occupancy
representation and distill the pre-trained open vocabulary model into a 3D
language field via volume rendering for zero-shot inference. Furthermore, a
novel semantic-aware confidence propagation (SCP) method has been proposed to
relieve the issue of language field representation degeneracy caused by
inconsistent measurements in distilled features. Experimental results show that
our approach achieves competitive performance in 3D scene understanding tasks,
especially for small and long-tail objects.
