  This paper proposes a differentiable robust LQR layer for reinforcement
learning and imitation learning under model uncertainty and stochastic
dynamics. The robust LQR layer can exploit the advantages of robust optimal
control and model-free learning. It provides a new type of inductive bias for
stochasticity and uncertainty modeling in control systems. In particular, we
propose an efficient way to differentiate through a robust LQR optimization
program by rewriting it as a convex program (i.e. semi-definite program) of the
worst-case cost. Based on recent work on using convex optimization inside
neural network layers, we develop a fully differentiable layer for optimizing
this worst-case cost, i.e. we compute the derivative of a performance measure
w.r.t the model's unknown parameters, model uncertainty and stochasticity
parameters. We demonstrate the proposed method on imitation learning and
approximate dynamic programming on stochastic and uncertain domains. The
experiment results show that the proposed method can optimize robust policies
under uncertain situations, and are able to achieve a significantly better
performance than existing methods that do not model uncertainty directly.
