  Robot manipulation of unknown objects in unstructured environments is a
challenging problem due to the variety of shapes, materials, arrangements and
lighting conditions. Even with large-scale real-world data collection, robust
perception and manipulation of transparent and reflective objects across
various lighting conditions remain challenging. To address these challenges we
propose an approach to performing sim-to-real transfer of robotic perception.
The underlying model, SimNet, is trained as a single multi-headed neural
network using simulated stereo data as input and simulated object segmentation
masks, 3D oriented bounding boxes (OBBs), object keypoints, and disparity as
output. A key component of SimNet is the incorporation of a learned stereo
sub-network that predicts disparity. SimNet is evaluated on 2D car detection,
unknown object detection, and deformable object keypoint detection and
significantly outperforms a baseline that uses a structured light RGB-D sensor.
By inferring grasp positions using the OBB and keypoint predictions, SimNet can
be used to perform end-to-end manipulation of unknown objects in both easy and
hard scenarios using our fleet of Toyota HSR robots in four home environments.
In unknown object grasping experiments, the predictions from the baseline RGB-D
network and SimNet enable successful grasps of most of the easy objects.
However, the RGB-D baseline only grasps 35% of the hard (e.g., transparent)
objects, while SimNet grasps 95%, suggesting that SimNet can enable robust
manipulation of unknown objects, including transparent objects, in unknown
environments.
