  This paper presents one of the first learning-based NeRF 3D instance
segmentation pipelines, dubbed as Instance Neural Radiance Field, or Instance
NeRF. Taking a NeRF pretrained from multi-view RGB images as input, Instance
NeRF can learn 3D instance segmentation of a given scene, represented as an
instance field component of the NeRF model. To this end, we adopt a 3D
proposal-based mask prediction network on the sampled volumetric features from
NeRF, which generates discrete 3D instance masks. The coarse 3D mask prediction
is then projected to image space to match 2D segmentation masks from different
views generated by existing panoptic segmentation models, which are used to
supervise the training of the instance field. Notably, beyond generating
consistent 2D segmentation maps from novel views, Instance NeRF can query
instance information at any 3D point, which greatly enhances NeRF object
segmentation and manipulation. Our method is also one of the first to achieve
such results in pure inference. Experimented on synthetic and real-world NeRF
datasets with complex indoor scenes, Instance NeRF surpasses previous NeRF
segmentation works and competitive 2D segmentation methods in segmentation
performance on unseen views. Watch the demo video at
https://youtu.be/wW9Bme73coI. Code and data are available at
https://github.com/lyclyc52/Instance_NeRF.
