  Perceiving a three-dimensional (3D) scene with multiple objects while moving
indoors is essential for vision-based mobile cobots, especially for enhancing
their manipulation tasks. In this work, we present an end-to-end pipeline with
instance segmentation, feature matching, and point-set registration for
egocentric robots with binocular vision, and demonstrate the robot's grasping
capability through the proposed pipeline. First, we design an RGB image-based
segmentation approach for single-view 3D semantic scene segmentation,
leveraging common object classes in 2D datasets to encapsulate 3D points into
point clouds of object instances through corresponding depth maps. Next, 3D
correspondences of two consecutive segmented point clouds are extracted based
on matched keypoints between objects of interest in RGB images from the prior
step. In addition, to be aware of spatial changes in 3D feature distribution,
we also weigh each 3D point pair based on the estimated distribution using
kernel density estimation (KDE), which subsequently gives robustness with less
central correspondences while solving for rigid transformations between point
clouds. Finally, we test our proposed pipeline on the 7-DOF dual-arm Baxter
robot with a mounted Intel RealSense D435i RGB-D camera. The result shows that
our robot can segment objects of interest, register multiple views while
moving, and grasp the target object. The source code is available at
https://github.com/mkhangg/semantic_scene_perception.
