  Nowadays, the mainstream approach in position allocation system is to utilize
a reinforcement learning model to allocate appropriate locations for items in
various channels and then mix them into the feed. There are two types of data
employed to train reinforcement learning (RL) model for position allocation,
named strategy data and random data. Strategy data is collected from the
current online model, it suffers from an imbalanced distribution of
state-action pairs, resulting in severe overestimation problems during
training. On the other hand, random data offers a more uniform distribution of
state-action pairs, but is challenging to obtain in industrial scenarios as it
could negatively impact platform revenue and user experience due to random
exploration. As the two types of data have different distributions, designing
an effective strategy to leverage both types of data to enhance the efficacy of
the RL model training has become a highly challenging problem. In this study,
we propose a framework named Multi-Distribution Data Learning (MDDL) to address
the challenge of effectively utilizing both strategy and random data for
training RL models on mixed multi-distribution data. Specifically, MDDL
incorporates a novel imitation learning signal to mitigate overestimation
problems in strategy data and maximizes the RL signal for random data to
facilitate effective learning. In our experiments, we evaluated the proposed
MDDL framework in a real-world position allocation system and demonstrated its
superior performance compared to the previous baseline. MDDL has been fully
deployed on the Meituan food delivery platform and currently serves over 300
million users.
