  In many reinforcement learning applications, the underlying environment
reward and transition functions are explicitly known differentiable functions.
This enables us to use recent research which applies machine learning tools to
stochastic control to find optimal action functions. In this paper, we define
differentiable reinforcement learning as a particular case of this research. We
find that incorporating deep learning in this framework leads to more accurate
and stable solutions than those obtained from more generic actor critic
algorithms. We apply this deep differentiable reinforcement learning (DDRL)
algorithm to the problem of one asset optimal trading strategies in various
environments where the market dynamics are known. Thanks to the stability of
this method, we are able to efficiently find optimal strategies for complex
multi-scale market models. We also extend these methods to simultaneously find
optimal action functions for a wide range of environment parameters. This makes
it applicable to real life financial signals and portfolio optimization where
the expected return has multiple time scales. In the case of a slow and a fast
alpha signal, we find that the optimal trading strategy consists in using the
fast signal to time the trades associated to the slow signal.
