  Exploration-Exploitation (E{\&}E) algorithms are commonly adopted to deal
with the feedback-loop issue in large-scale online recommender systems. Most of
existing studies believe that high uncertainty can be a good indicator of
potential reward, and thus primarily focus on the estimation of model
uncertainty. We argue that such an approach overlooks the subsequent effect of
exploration on model training. From the perspective of online learning, the
adoption of an exploration strategy would also affect the collecting of
training data, which further influences model learning. To understand the
interaction between exploration and training, we design a Pseudo-Exploration
module that simulates the model updating process after a certain item is
explored and the corresponding feedback is received. We further show that such
a process is equivalent to adding an adversarial perturbation to the model
input, and thereby name our proposed approach as an the Adversarial Gradient
Driven Exploration (AGE). For production deployment, we propose a dynamic
gating unit to pre-determine the utility of an exploration. This enables us to
utilize the limited amount of resources for exploration, and avoid wasting
pageview resources on ineffective exploration. The effectiveness of AGE was
firstly examined through an extensive number of ablation studies on an academic
dataset. Meanwhile, AGE has also been deployed to one of the world-leading
display advertising platforms, and we observe significant improvements on
various top-line evaluation metrics.
