  Designing an intelligent volume-weighted average price (VWAP) strategy is a
critical concern for brokers, since traditional rule-based strategies are
relatively static that cannot achieve a lower transaction cost in a dynamic
market. Many studies have tried to minimize the cost via reinforcement
learning, but there are bottlenecks in improvement, especially for
long-duration strategies such as the VWAP strategy. To address this issue, we
propose a deep learning and hierarchical reinforcement learning jointed
architecture termed Macro-Meta-Micro Trader (M3T) to capture market patterns
and execute orders from different temporal scales. The Macro Trader first
allocates a parent order into tranches based on volume profiles as the
traditional VWAP strategy does, but a long short-term memory neural network is
used to improve the forecasting accuracy. Then the Meta Trader selects a
short-term subgoal appropriate to instant liquidity within each tranche to form
a mini-tranche. The Micro Trader consequently extracts the instant market state
and fulfils the subgoal with the lowest transaction cost. Our experiments over
stocks listed on the Shanghai stock exchange demonstrate that our approach
outperforms baselines in terms of VWAP slippage, with an average cost saving of
1.16 base points compared to the optimal baseline.
