  Dexterous manipulation, often facilitated by multi-fingered robotic hands,
holds solid impact for real-world applications. Soft robotic hands, due to
their compliant nature, offer flexibility and adaptability during object
grasping and manipulation. Yet, benefits come with challenges, particularly in
the control development for finger coordination. Reinforcement Learning (RL)
can be employed to train object-specific in-hand manipulation policies, but
limiting adaptability and generalizability. We introduce a Continual Policy
Distillation (CPD) framework to acquire a versatile controller for in-hand
manipulation, to rotate different objects in shape and size within a
four-fingered soft gripper. The framework leverages Policy Distillation (PD) to
transfer knowledge from expert policies to a continually evolving student
policy network. Exemplar-based rehearsal methods are then integrated to
mitigate catastrophic forgetting and enhance generalization. The performance of
the CPD framework over various replay strategies demonstrates its effectiveness
in consolidating knowledge from multiple experts and achieving versatile and
adaptive behaviours for in-hand manipulation tasks.
