  Optimal control (OC) algorithms such as Differential Dynamic Programming
(DDP) take advantage of the derivatives of the dynamics to efficiently control
physical systems. Yet, in the presence of nonsmooth dynamical systems, such
class of algorithms are likely to fail due, for instance, to the presence of
discontinuities in the dynamics derivatives or because of non-informative
gradient. On the contrary, reinforcement learning (RL) algorithms have shown
better empirical results in scenarios exhibiting non-smooth effects (contacts,
frictions, etc). Our approach leverages recent works on randomized smoothing
(RS) to tackle non-smoothness issues commonly encountered in optimal control,
and provides key insights on the interplay between RL and OC through the prism
of RS methods. This naturally leads us to introduce the randomized Differential
Dynamic Programming (R-DDP) algorithm accounting for deterministic but
non-smooth dynamics in a very sample-efficient way. The experiments demonstrate
that our method is able to solve classic robotic problems with dry friction and
frictional contacts, where classical OC algorithms are likely to fail and RL
algorithms require in practice a prohibitive number of samples to find an
optimal solution.
