  Manipulation relationship detection (MRD) aims to guide the robot to grasp
objects in the right order, which is important to ensure the safety and
reliability of grasping in object stacked scenes. Previous works infer
manipulation relationship by deep neural network trained with data collected
from a predefined view, which has limitation in visual dislocation in
unstructured environments. Multi-view data provide more comprehensive
information in space, while a challenge of multi-view MRD is domain shift. In
this paper, we propose a novel multi-view fusion framework, namely multi-view
MRD network (MMRDN), which is trained by 2D and 3D multi-view data. We project
the 2D data from different views into a common hidden space and fit the
embeddings with a set of Von-Mises-Fisher distributions to learn the consistent
representations. Besides, taking advantage of position information within the
3D data, we select a set of $K$ Maximum Vertical Neighbors (KMVN) points from
the point cloud of each object pair, which encodes the relative position of
these two objects. Finally, the features of multi-view 2D and 3D data are
concatenated to predict the pairwise relationship of objects. Experimental
results on the challenging REGRAD dataset show that MMRDN outperforms the
state-of-the-art methods in multi-view MRD tasks. The results also demonstrate
that our model trained by synthetic data is capable to transfer to real-world
scenarios.
