  The emergence of differentiable simulators enabling analytic gradient
computation has motivated a new wave of learning algorithms that hold the
potential to significantly increase sample efficiency over traditional
Reinforcement Learning (RL) methods. While recent research has demonstrated
performance gains in scenarios with comparatively smooth dynamics and, thus,
smooth optimization landscapes, research on leveraging differentiable
simulators for contact-rich scenarios, such as legged locomotion, is scarce.
This may be attributed to the discontinuous nature of contact, which introduces
several challenges to optimizing with analytic gradients. The purpose of this
paper is to determine if analytic gradients can be beneficial even in the face
of contact. Our investigation focuses on the effects of different soft and hard
contact models on the learning process, examining optimization challenges
through the lens of contact simulation. We demonstrate the viability of
employing analytic gradients to learn physically plausible locomotion skills
with a quadrupedal robot using Short-Horizon Actor-Critic (SHAC), a learning
algorithm leveraging analytic gradients, and draw a comparison to a
state-of-the-art RL algorithm, Proximal Policy Optimization (PPO), to
understand the benefits of analytic gradients.
