  In this work, we aim to learn dexterous manipulation of deformable objects
using multi-fingered hands. Reinforcement learning approaches for dexterous
rigid object manipulation would struggle in this setting due to the complexity
of physics interaction with deformable objects. At the same time, previous
trajectory optimization approaches with differentiable physics for deformable
manipulation would suffer from local optima caused by the explosion of contact
modes from hand-object interactions. To address these challenges, we propose
DexDeform, a principled framework that abstracts dexterous manipulation skills
from human demonstration and refines the learned skills with differentiable
physics. Concretely, we first collect a small set of human demonstrations using
teleoperation. And we then train a skill model using demonstrations for
planning over action abstractions in imagination. To explore the goal space, we
further apply augmentations to the existing deformable shapes in demonstrations
and use a gradient optimizer to refine the actions planned by the skill model.
Finally, we adopt the refined trajectories as new demonstrations for finetuning
the skill model. To evaluate the effectiveness of our approach, we introduce a
suite of six challenging dexterous deformable object manipulation tasks.
Compared with baselines, DexDeform is able to better explore and generalize
across novel goals unseen in the initial human demonstrations.
