  This paper introduces LeTO, a method for learning constrained visuomotor
policy via differentiable trajectory optimization. Our approach uniquely
integrates a differentiable optimization layer into the neural network. By
formulating the optimization layer as a trajectory optimization problem, we
enable the model to end-to-end generate actions in a safe and controlled
fashion without extra modules. Our method allows for the introduction of
constraints information during the training process, thereby balancing the
training objectives of satisfying constraints, smoothing the trajectories, and
minimizing errors with demonstrations. This "gray box" method marries the
optimization-based safety and interpretability with the powerful
representational abilities of neural networks. We quantitatively evaluate LeTO
in simulation and on the real robot. In simulation, LeTO achieves a success
rate comparable to state-of-the-art imitation learning methods, but the
generated trajectories are of less uncertainty, higher quality, and smoother.
In real-world experiments, we deployed LeTO to handle constraints-critical
tasks. The results show the effectiveness of LeTO comparing with
state-of-the-art imitation learning approaches. We release our code at
https://github.com/ZhengtongXu/LeTO.
