  High-frequency trading (HFT) that executes algorithmic trading in short time
scales, has recently occupied the majority of cryptocurrency market. Besides
traditional quantitative trading methods, reinforcement learning (RL) has
become another appealing approach for HFT due to its terrific ability of
handling high-dimensional financial data and solving sophisticated sequential
decision-making problems, \emph{e.g.,} hierarchical reinforcement learning
(HRL) has shown its promising performance on second-level HFT by training a
router to select only one sub-agent from the agent pool to execute the current
transaction. However, existing RL methods for HFT still have some defects: 1)
standard RL-based trading agents suffer from the overfitting issue, preventing
them from making effective policy adjustments based on financial context; 2)
due to the rapid changes in market conditions, investment decisions made by an
individual agent are usually one-sided and highly biased, which might lead to
significant loss in extreme markets. To tackle these problems, we propose a
novel Memory Augmented Context-aware Reinforcement learning method On HFT,
\emph{a.k.a.} MacroHFT, which consists of two training phases: 1) we first
train multiple types of sub-agents with the market data decomposed according to
various financial indicators, specifically market trend and volatility, where
each agent owns a conditional adapter to adjust its trading policy according to
market conditions; 2) then we train a hyper-agent to mix the decisions from
these sub-agents and output a consistently profitable meta-policy to handle
rapid market fluctuations, equipped with a memory mechanism to enhance the
capability of decision-making. Extensive experiments on various cryptocurrency
markets demonstrate that MacroHFT can achieve state-of-the-art performance on
minute-level trading tasks.
