  Optimal order execution is widely studied by industry practitioners and
academic researchers because it determines the profitability of investment
decisions and high-level trading strategies, particularly those involving large
volumes of orders. However, complex and unknown market dynamics pose
significant challenges for the development and validation of optimal execution
strategies. In this paper, we propose a model-free approach by training
Reinforcement Learning (RL) agents in a realistic market simulation environment
with multiple agents. First, we configure a multi-agent historical order book
simulation environment for execution tasks built on an Agent-Based Interactive
Discrete Event Simulation (ABIDES) [arXiv:1904.12066]. Second, we formulate the
problem of optimal execution in an RL setting where an intelligent agent can
make order execution and placement decisions based on market microstructure
trading signals in High Frequency Trading (HFT). Third, we develop and train an
RL execution agent using the Double Deep Q-Learning (DDQL) algorithm in the
ABIDES environment. In some scenarios, our RL agent converges towards a
Time-Weighted Average Price (TWAP) strategy. Finally, we evaluate the
simulation with our RL agent by comparing it with a market replay simulation
using real market Limit Order Book (LOB) data.
