  Object manipulation from 3D visual inputs poses many challenges on building
generalizable perception and policy models. However, 3D assets in existing
benchmarks mostly lack the diversity of 3D shapes that align with real-world
intra-class complexity in topology and geometry. Here we propose SAPIEN
Manipulation Skill Benchmark (ManiSkill) to benchmark manipulation skills over
diverse objects in a full-physics simulator. 3D assets in ManiSkill include
large intra-class topological and geometric variations. Tasks are carefully
chosen to cover distinct types of manipulation challenges. Latest progress in
3D vision also makes us believe that we should customize the benchmark so that
the challenge is inviting to researchers working on 3D deep learning. To this
end, we simulate a moving panoramic camera that returns ego-centric point
clouds or RGB-D images. In addition, we would like ManiSkill to serve a broad
set of researchers interested in manipulation research. Besides supporting the
learning of policies from interactions, we also support
learning-from-demonstrations (LfD) methods, by providing a large number of
high-quality demonstrations (~36,000 successful trajectories, ~1.5M point
cloud/RGB-D frames in total). We provide baselines using 3D deep learning and
LfD algorithms. All code of our benchmark (simulator, environment, SDK, and
baselines) is open-sourced, and a challenge facing interdisciplinary
researchers will be held based on the benchmark.
