  We address the important problem of generalizing robotic rearrangement to
clutter without any explicit object models. We first generate over 650K
cluttered scenes - orders of magnitude more than prior work - in diverse
everyday environments, such as cabinets and shelves. We render synthetic
partial point clouds from this data and use it to train our CabiNet model
architecture. CabiNet is a collision model that accepts object and scene point
clouds, captured from a single-view depth observation, and predicts collisions
for SE(3) object poses in the scene. Our representation has a fast inference
speed of 7 microseconds per query with nearly 20% higher performance than
baseline approaches in challenging environments. We use this collision model in
conjunction with a Model Predictive Path Integral (MPPI) planner to generate
collision-free trajectories for picking and placing in clutter. CabiNet also
predicts waypoints, computed from the scene's signed distance field (SDF), that
allows the robot to navigate tight spaces during rearrangement. This improves
rearrangement performance by nearly 35% compared to baselines. We
systematically evaluate our approach, procedurally generate simulated
experiments, and demonstrate that our approach directly transfers to the real
world, despite training exclusively in simulation. Robot experiment demos in
completely unknown scenes and objects can be found at this http
https://cabinet-object-rearrangement.github.io
