User Tampering in Reinforcement Learning Recommender Systems
  This paper provides the first formalisation and empirical demonstration of a
particular safety concern in reinforcement learning (RL)-based news and social
media recommendation algorithms. This safety concern is what we call "user
tampering" -- a phenomenon whereby an RL-based recommender system may
manipulate a media user's opinions, preferences and beliefs via its
recommendations as part of a policy to increase long-term user engagement. We
provide a simulation study of a media recommendation problem constrained to the
recommendation of political content, and demonstrate that a Q-learning
algorithm consistently learns to exploit its opportunities to 'polarise'
simulated 'users' with its early recommendations in order to have more
consistent success with later recommendations catering to that polarisation.
Finally, we argue that given our findings, designing an RL-based recommender
system which cannot learn to exploit user tampering requires making the metric
for the recommender's success independent of observable signals of user
engagement, and thus that a media recommendation system built solely with RL is
necessarily either unsafe, or almost certainly commercially unviable.
