  In this work, we tackle 6-DoF grasp detection for transparent and specular
objects, which is an important yet challenging problem in vision-based robotic
systems, due to the failure of depth cameras in sensing their geometry. We, for
the first time, propose a multiview RGB-based 6-DoF grasp detection network,
GraspNeRF, that leverages the generalizable neural radiance field (NeRF) to
achieve material-agnostic object grasping in clutter. Compared to the existing
NeRF-based 3-DoF grasp detection methods that rely on densely captured input
images and time-consuming per-scene optimization, our system can perform
zero-shot NeRF construction with sparse RGB inputs and reliably detect 6-DoF
grasps, both in real-time. The proposed framework jointly learns generalizable
NeRF and grasp detection in an end-to-end manner, optimizing the scene
representation construction for the grasping. For training data, we generate a
large-scale photorealistic domain-randomized synthetic dataset of grasping in
cluttered tabletop scenes that enables direct transfer to the real world. Our
extensive experiments in synthetic and real-world environments demonstrate that
our method significantly outperforms all the baselines in all the experiments
while remaining in real-time. Project page can be found at
https://pku-epic.github.io/GraspNeRF
