  We argue that inventory management presents unique opportunities for reliably
applying and evaluating deep reinforcement learning (DRL). Toward reliable
application, we emphasize and test two techniques. The first is Hindsight
Differentiable Policy Optimization (HDPO), which performs stochastic gradient
descent to optimize policy performance while avoiding the need to repeatedly
deploy randomized policies in the environment-as is common with generic policy
gradient methods. Our second technique involves aligning policy (neural)
network architectures with the structure of the inventory network.
Specifically, we focus on a network with a single warehouse that consolidates
inventory from external suppliers, holds it, and then distributes it to many
stores as needed. In this setting, we introduce the symmetry-aware policy
network architecture. We motivate this architecture by establishing an
asymptotic performance guarantee and empirically demonstrate its ability to
reduce the amount of data needed to uncover strong policies. Both techniques
exploit structures inherent in inventory management problems, moving beyond
generic DRL algorithms. Toward rigorous evaluation, we create and share new
benchmark problems, divided into two categories. One type focuses on problems
with hidden structures that allow us to compute or bound the cost of the true
optimal policy. Across four problems of this type, we find HDPO consistently
attains near-optimal performance, handling up to 60-dimensional raw state
vectors effectively. The other type of evaluation involves constructing a test
problem using real time series data from a large retailer, where the optimum is
poorly understood. Here, we find HDPO methods meaningfully outperform a variety
of generalized newsvendor heuristics. Our code can be found at
github.com/MatiasAlvo/Neural_inventory_control.
