  Adversarial attacks in reinforcement learning (RL) often assume
highly-privileged access to the victim's parameters, environment, or data.
Instead, this paper proposes a novel adversarial setting called a Cheap Talk
MDP in which an Adversary can merely append deterministic messages to the
Victim's observation, resulting in a minimal range of influence. The Adversary
cannot occlude ground truth, influence underlying environment dynamics or
reward signals, introduce non-stationarity, add stochasticity, see the Victim's
actions, or access their parameters. Additionally, we present a simple
meta-learning algorithm called Adversarial Cheap Talk (ACT) to train
Adversaries in this setting. We demonstrate that an Adversary trained with ACT
still significantly influences the Victim's training and testing performance,
despite the highly constrained setting. Affecting train-time performance
reveals a new attack vector and provides insight into the success and failure
modes of existing RL algorithms. More specifically, we show that an ACT
Adversary is capable of harming performance by interfering with the learner's
function approximation, or instead helping the Victim's performance by
outputting useful features. Finally, we show that an ACT Adversary can
manipulate messages during train-time to directly and arbitrarily control the
Victim at test-time. Project video and code are available at
https://sites.google.com/view/adversarial-cheap-talk
