  Inspired by the developments in deep generative models, we propose a
model-based RL approach, coined Reinforced Deep Markov Model (RDMM), designed
to integrate desirable properties of a reinforcement learning algorithm acting
as an automatic trading system. The network architecture allows for the
possibility that market dynamics are partially visible and are potentially
modified by the agent's actions. The RDMM filters incomplete and noisy data, to
create better-behaved input data for RL planning. The policy search
optimisation also properly accounts for state uncertainty. Due to the
complexity of the RKDF model architecture, we performed ablation studies to
understand the contributions of individual components of the approach better.
To test the financial performance of the RDMM we implement policies using
variants of Q-Learning, DynaQ-ARIMA and DynaQ-LSTM algorithms. The experiments
show that the RDMM is data-efficient and provides financial gains compared to
the benchmarks in the optimal execution problem. The performance improvement
becomes more pronounced when price dynamics are more complex, and this has been
demonstrated using real data sets from the limit order book of Facebook, Intel,
Vodafone and Microsoft.
