  In retail (e.g., grocery stores, apparel shops, online retailers), inventory
managers have to balance short-term risk (no items to sell) with long-term-risk
(over ordering leading to product waste). This balancing task is made
especially hard due to the lack of information about future customer purchases.
In this paper, we study the problem of restocking a grocery store's inventory
with perishable items over time, from a distributional point of view. The
objective is to maximize sales while minimizing waste, with uncertainty about
the actual consumption by costumers. This problem is of a high relevance today,
given the growing demand for food and the impact of food waste on the
environment, the economy, and purchasing power. We frame inventory restocking
as a new reinforcement learning task that exhibits stochastic behavior
conditioned on the agent's actions, making the environment partially
observable. We make two main contributions. First, we introduce a new
reinforcement learning environment, RetaiL, based on real grocery store data
and expert knowledge. This environment is highly stochastic, and presents a
unique challenge for reinforcement learning practitioners. We show that
uncertainty about the future behavior of the environment is not handled well by
classical supply chain algorithms, and that distributional approaches are a
good way to account for the uncertainty. Second, we introduce GTDQN, a
distributional reinforcement learning algorithm that learns a generalized Tukey
Lambda distribution over the reward space. GTDQN provides a strong baseline for
our environment. It outperforms other distributional reinforcement learning
approaches in this partially observable setting, in both overall reward and
reduction of generated waste.
