Disturbing Reinforcement Learning Agents \\ with Corrupted Rewards
Rubén Majadas
Javier García
Fernando Fernández\institute{This paper has been accepted in RAISA3 workshop celebrated in ECAI 2020 conference. Departamento de Informática
Universidad Carlos III de Madrid Avda. de la Universidad
30. 28911 Leganés (Madrid). Spain Email addresses: rmajadas@pa.uc3m.es (Rubén Majadas)
fjgpolo@inf.uc3m.es (Javier García) and ffernand@inf.uc3m.es (Fernando Fernández)
%Quitar un goal-only-rewards si introducimos refuerzos intermedio Reinforcement Learning (RL) algorithms have led to recent successes in solving complex games, such as Atari or Starcraft, and to a huge impact in real-world applications, such as cybersecurity or autonomous driving. In the side of the drawbacks, recent works have shown how the performance of RL algorithms decreases under the influence of soft changes in the reward function. However, little work has been done about how sensitive these disturbances are depending on the aggressiveness of the attack and the learning exploration strategy. In this paper, we propose to fill this gap in the literature analyzing the effects of different attack strategies based on reward perturbations, and studying the effect in the learner depending on its exploration strategy. In order to explain all the behaviors, we choose a sub-class of MDPs: episodic, stochastic goal-only-rewards MDPs, and in particular, an intelligible grid domain as a benchmark. In this domain, we demonstrate that smoothly crafting adversarial rewards are able to mislead the learner, and that using low exploration probability values, the policy learned is more robust to corrupt rewards. Finally, in the proposed learning scenario, a counterintuitive result arises: attacking at each learning episode is the lowest cost attack strategy. %\keywords{Adversarial RL \and Adversarial rewards \and Exploration strategies
main.tex
% This is samplepaper.tex, a sample chapter demonstrating the
% LLNCS macro package for Springer Computer Science proceedings;
% Version 2.20 of 2017/10/04
%
\documentclass{ecai}
%
\usepackage{graphicx}
\usepackage{xspace}
\usepackage{algorithm}
\usepackage[noend]{algorithmic}
\usepackage{amsmath}
\usepackage{float}

% Used for displaying a sample figure. If possible, figure files should
% be included in EPS format.
%
% If you use the hyperref package, please uncomment the following line
% to display URLs in blue roman font according to Springer's eBook style:
% \renewcommand\UrlFont{\color{blue}\rmfamily}

\ecaisubmission

\begin{document}
%
\title{Disturbing Reinforcement Learning Agents \\ with Corrupted Rewards}
\author{Rubén Majadas \and Javier García  \and Fernando Fernández\institute{This paper has been accepted in RAISA3 workshop celebrated in ECAI 2020 conference. Departamento de Informática, Universidad Carlos III de Madrid Avda. de la Universidad, 30. 28911 Leganés (Madrid). Spain Email addresses: rmajadas@pa.uc3m.es (Rubén Majadas), fjgpolo@inf.uc3m.es (Javier García) and ffernand@inf.uc3m.es (Fernando Fernández)}}



\maketitle              % typeset the header of the contribution
%
\begin{abstract}
%Quitar un goal-only-rewards si introducimos refuerzos intermedio
Reinforcement Learning (RL) algorithms have led to recent successes in solving complex games, such as Atari or Starcraft, and to a huge impact in real-world applications, such as cybersecurity or autonomous driving. In the side of the drawbacks, recent works have shown how the performance of RL algorithms decreases under the influence of soft changes in the reward function. However, little work has been done about how sensitive these disturbances are depending on the aggressiveness of the attack and the learning exploration strategy. In this paper, we propose to fill this gap in the literature analyzing the effects of different attack strategies based on reward perturbations, and studying the effect in the learner depending on its exploration strategy. In order to explain all the behaviors, we choose a sub-class of MDPs: episodic, stochastic \textit{goal-only-rewards} MDPs, and in particular, an intelligible grid domain as a benchmark. In this domain, we demonstrate that smoothly crafting adversarial rewards are able to mislead the learner, and that using low exploration probability values, the policy learned is more robust to corrupt rewards. Finally, in the proposed learning scenario, a counterintuitive result arises: attacking at each learning episode is the lowest cost attack strategy.

%\keywords{Adversarial RL  \and Adversarial rewards \and Exploration strategies}
\end{abstract}
%
%
%
\section{Introduction}

Reinforcement Learning (RL) is a type of machine learning whose main goal is finding an action policy that makes the agent act optimally in an environment, generally formulated as a Markov Decision Process (MDP). Recently, RL has been successfully used in complex computer games (e.g., Atari~\cite{mnih2013playing}, or Starcraft~\cite{alphastarblog2}), but also in real-world applications where robustness is a key factor (e.g., robotic tasks~\cite{DBLP:journals/jmlr/LevineFDA16}, cyber security~\cite{ferdowsi2018robust} or autonomous driving~\cite{kuderer2015learning}). However, the robustness of RL algorithms is in doubt due to recent works showing that they are vulnerable to \textit{adversarial attacks}. In Adversarial RL~\cite{tong2019}, the most usual of these attacks tries to slightly change the observation (i.e., the state) that the agent perceives from the environment~\cite{behzadan2017vulnerability}, similarly to adversarial examples for classifiers~\cite{DBLP:journals/corr/SzegedyZSBEGF13}. Such attacks can be carried out during the training~\cite{kos2017delving} or the test phase~\cite{DBLP:journals/corr/HuangPGDA17}, and the objective is to disturb as little as possible the original state so as not to be detected, while achieving a system failure, i.e., to make the agent believe that it is in a situation when it is really in another one.

Besides the state perception, another sensitive target to attack is the reward function. In real-world applications, we can obtain rewards that differ from the original due to several factors such as incorrect measurements from sensors, noise in the transmission channel, but also due to attacks that intentionally perturb the reward signal.  However, this attack source has received very little attention~\cite{romoff2018reward,DBLP:journals/corr/abs-1810-01032}, and as far as we know, there are no studies about how sensitive the learning process is depending on the aggressiveness of the reward perturbations and the exploration strategy. In spite of that, we consider that such an analysis is essential for everyone who works in the emerging area of Adversarial RL both designing attacks and defense mechanisms. 

In this paper, we contribute to fill this gap in the literature. In particular, we address a sub-class of MDPs: episodic, stochastic \textit{goal-only-rewards} MDPs~\cite{reinke2017}. For \textit{goal-only-reward} MDPs, a reward is only given if a goal state is reached. Although this is a strict restriction, many decision tasks are modeled with \textit{goal-only-rewards} from classical control problems such as the \textit{Mountain Car} or the \textit{Cart Pole}, to more complex problems such as real robot manipulation tasks~\cite{morere18a}. However, we focus on an intelligible 20$\times$20 grid domain with goal states and obstacles. Focusing on an on-policy RL algorithm, SARSA, and two well-known exploration strategies ($\epsilon$-greedy  and softmax), this paper address a primary question: \textit{how the disturbances in the reward function when a goal state is reached affect the convergence of a learning agent depending on the exploration strategy selected?}

Therefore, our contribution is twofold: first, a threat model for the reward function in \textit{goal-only-rewards} MDPs; second, a comprehensive study in order to analyze the robustness of the learning process depending on the probability of reward perturbations and the exploration strategy. Results show that small exploration probabilities are more resilient to adversary presence, and that attacking at each learning episode is the lowest cost attack strategy.


\begin{section}{Background}


In the literature, we identify three different \textit{attack targets} where one can inject Adversarial Examples in any RL algorithm. 
Figure~\ref{fig:RLproccess} shows these attack targets where each adversary represents a different kind of attack. These possible attack points are the state perceived, the action selected or the reward function:
\begin{figure}[htb!]
    \centering
    \includegraphics[width=0.9\columnwidth]{RLdiagram.png}
    \caption{RL proccess during training phase. Each adversary represents a target where the RL training process can be perturbed.}
    \label{fig:RLproccess}
\end{figure}
\begin{enumerate}
    \item \textbf{State perception.} Most works focus on attacking the state perception in training \cite{kos2017delving,pattanaik2018robust} or testing \cite{DBLP:journals/corr/HuangPGDA17,pinto2017robust}. All these works have the common objective of delaying, perturbing or falsifying under malicious attacks the state $s$ perceived by the learning agent. 
    
    \item \textbf{Action selected.} There are also actuator attacks where the objective is to drive the learner agent to undesired states by perturbing at each time step the action $a$ executed by the learning agent~\cite{roy2017reinforcement}.
    
    \item \textbf{Training reward.} Finally, there are reward attacks. These attacks refer to the perturbation of the reward produced by the environment in response to the actions applied by a RL agent~\cite{Everitt2017rc}. 
    
\end{enumerate}
%%FFR He cambiado algunas cosas par no usar referencias como sustantivos. Lo he hecho ev varios, Rubén, termínalo.

This paper is included in the last category. However, very few works have explicitly studied the modification of the reward function in order to mislead the agent. In one of the first works ~\cite{Everitt2017rc}, the authors propose to modify a set of states where the perceived reward is  different from the real. Then, if the agent reaches these states the enemy performs an attack. Unlike our approach, we establish an attack probability just when the agent achieves a goal state, which significantly reduces the number of attacks. Another possibility is to generate surrogate rewards~\cite{DBLP:journals/corr/abs-1810-01032} from previous noisy rewards to train its agent avoiding the interference of the corruption. Because of this, the learning agent is unable to find the right policy, but also requires a large number of attacks. Instead, modifying the reward with a certain probability following some probability distribution~\cite{romoff2018reward} reduces the attack cost. But in the case where the agent learns without enemy presence, the agent receives wrong reward estimators. Therefore, its performance decreases in a free adversary scenario.  
Another novel approach also modifies rewards driving the learner to the desired goal ~\cite{ma2019policy}. However, unlike our approach, they need more knowledge about the problem, the whole training dataset, and the actions the learner can perform. 
Recently, a framework to perform offline and online attacks on the Stochastic multi-armed bandit problem has been published  ~\cite{liu2019data}. The adversary produces a linear regret on the bandit algorithm with only a logarithmic cost applying their adaptive attack strategy.  As in this approach, our work investigates how to reduce the attacking cost. Therefore, compared to the works mentioned above, our work has a completely different objective: we develop a comprehensive study to characterize how the perturbation probabilities and exploration strategies impact on the convergence of the learning agent in order to find the best configuration from the point of view of the learner but also the adversary. As far as we know, such study has never been carried out before.
    
\end{section}


\begin{section}{Problem Formulation}

We define this task as a stochastic \textit{goal-only-rewards} MDP that is represented by the tuple: (${\cal S}$ , ${\cal A}$, ${\cal G} $, ${\cal P}$, ${r}$, ${\cal \gamma}$, ${s_0}$), where ${\cal S}$ and ${\cal A}$ are state spaces of states $s$ and actions $a$ respectively, and $\cal G \subset S$ is the set of goal states. In the simplest case, \textit{goal-only rewards} are given for states $s \in \cal G$, and zero-reward is given elsewhere. ${\cal P}$ $ : $ ${\cal S} \times {\cal A} \times {\cal S} \rightarrow   {\cal R}$ is the unknown transition probability, $r : {\cal S} \times {\cal A} \rightarrow {\cal R}$ is the reward function, also unknown. $\gamma$ is the discount factor and $s_0$ is the initial state. In our study, we use an on-policy RL algorithm, SARSA. This on-policy algorithm learns the $Q$-value based in the next action performed by the current policy instead of taking the best action as the $Q$-learning algorithm. We update the $Q$ table values with the following function:

\begin{equation}
%\small
Q\left(s_{t}, a_{t}\right) \leftarrow(1-\alpha)^{*} Q\left(s_{t}, a_{t}\right)+\alpha\left[r_{t}+\gamma Q\left(s_{t+1}, a_{t+1}\right)\right]
\end{equation}

\noindent where $s_t \in {\cal S}$ represents the current state, $a_t \in {\cal A}$ is the current action performed, $r_t$ is the reward received in this time-step, $s_{t+1} \in {\cal S}$ is the next state and $a_{t+1} \in {\cal A}$ is the action performed in the next state. $\alpha$ is the learning rate.
\subsection{Exploration Strategies and Learner Objective}
In this paper, we study two exploration/exploitation strategies: $\epsilon$-greedy and softmax. $\epsilon$-greedy chooses the best action with a probability $1 - \epsilon$, and a random action with probability $\epsilon$, as shown in Equation~\ref{eq:epsilon}.

\begin{equation}
A \leftarrow\left\{\begin{array}{ll}{\operatorname{argmax}_{a} Q(s, a)} & {\text { with probability } 1-\epsilon} \\ {\text {a random action }} & {\text { with probability } \epsilon}\end{array}\right.
\label{eq:epsilon}
\end{equation}

Instead, softmax assigns a probability of choosing each action as described in Equation~\ref{eq:softmax}. 

\begin{equation}
P(a) = \frac{e^{Q_{t}(a) / \tau}}{\sum_{b=1}^{n} e^{Q_{t}(b) / \tau}}
\label{eq:softmax}
\end{equation}

In Equation~\ref{eq:softmax}, $\tau$ is the temperature parameter. If the value of this parameter is close to 0, the agent will behave totally greedy. On the contrary, if this value is greater, all the actions will be equally probable of performing it.

The goal is to maximize the expected discounted reward per episode as described in Equation~\ref{eq:metric}:
%% FFR: la he modificado para meter la esperanza

\begin{equation}
\label{eq:metric}
\max_{\pi} (\sum_{t=0}^{N} \gamma^{t} r^{t}) 
%%= \max_{\pi} \gamma^{N} r^{N} Creo que esta parte confunde. La cuestión es que N, definido como pasos hasta la meta, cambia en cada episodio. Y matemáticamente no es adecuado.
\end{equation}

\noindent where $\gamma$ is the discount factor, and $N$ is the maximum number of steps allowed in an episode. It is important to note that this discounted reward per episode will be zero if a goal state is not reached.

\subsection{Threat Model}

We assume that the adversary does not have information about the learner (black-box attack). The adversary can only create bounded perturbations to the true reward signal when the learner reaches a goal state, i.e.,  $\forall s \in \cal G$, $r \neq 0$. In this case, the action of the adversary is limited to changing the sign of the true reward as shown in Equation~\ref{eq:inverseReward}:

\begin{equation}
\label{eq:inverseReward}
    \hat{r}= - r
\end{equation}

\noindent where $r$ is the true reward perceived from the environment, and $\hat{r}$ is the corrupt reward.


Algorithm~\ref{algorithm} describes how the adversary operates during the learning process. The adversary receives as input parameter an attack probability, $p$. Then, it is continuously sniffing the reward channel between the environment and the learning agent as described in Figure~\ref{fig:RLproccess} (line 2 of Algorithm~\ref{algorithm}). If the perceived reward corresponds to a goal state, i.e., $r \neq 0$ (line 3), then the adversary generates a random number $\varphi \in [0,1]$ (line 5). If $\varphi$ is less or equal to $p$, the adversary injects a corrupt reward in the reward channel (lines 6-7). Therefore, at each time $t$ the learning agent reaches a goal state, instead of observing $r$ directly, the agent only observes with probability $p$ a falsified reward $\hat{r}$, ignoring the presence of the adversary.

\begin{algorithm}
\caption{Enemy Injection Algorithm}
\label{algorithm}
\begin{algorithmic}[1]
 \REQUIRE $p$ 
 \REPEAT
 \STATE $r \leftarrow$ \textit{SniffingRewardChannel() }
 \IF{$r \neq 0$} 
 \STATE $\varphi \leftarrow rand() $ 
 \IF{$\varphi \leq p$}
 \STATE $\hat{r} \leftarrow - r$
 \STATE $inject(\hat{r})$
 \ENDIF
 \ENDIF
  \UNTIL stop criterion becomes true
 \RETURN 
\end{algorithmic}
\end{algorithm}

In the experiments, we will evaluate the performance of the adversary according to two criteria. First, the performance of the learning agent according to Equation~\ref{eq:metric}, and then, we count the number of attacks performed. Therefore, the worse the learning agent performance and the lower the number of attacks, the better adversary behavior.


\end{section}


\begin{section}{Evaluation}

The experiments aim to study the behavior of a learning agent and an adversary in a \textit{goal-only reward} MDP. The following sections describe the evaluation environment, the evaluation settings, and the results achieved.

\subsection{Evaluation Environment}


Figure \ref{fig:map} shows an example random 20x20 grid used to perform the experiments. We run a random scenario generator to create different learning episodes.  The green cell denotes the initial state where the learning agent is starting, the red cells denote the goals  it has to reach, and the brown cells denote obstacles. The agent has the actions $\cal{A}=\{\uparrow, \downarrow, \leftarrow, \rightarrow\}$. An action $a \in \cal{A}$ takes the agent in the denoted direction if possible. With a probability of 0.1, the agent is not transported to the desired direction but to one of the three remaining directions. In total, we generate 50 different grids placing the goal states at a similar distance from the starting point. We put these goals between 20 and 25 steps from the initial state.
The results show the average of all the scenarios. 


\begin{figure}[ht!]
    \centering
    \includegraphics[scale=0.4]{map-example.png}
    \caption{Random map example}
    \label{fig:map}
\end{figure}


%Cambiar la funcion de refuerzo
The learning agent starts in the initial position and it can move until one of the goal states or until a maximum number number of steps per episode is reached. We establish this maximum number of steps to 500, a enough number of steps to allow the agent to explore and to reach one goal considering the size of the grid. We establish a different reward for each goal $r = \{0.25,\; 0.5\; or \; 1\}$, and $r=0$ otherwise. 



\subsection{Evaluation Settings}

We have performed two sets of experiments. In our first experiments, both exploration strategies fix the probability of exploring and exploiting. For the second set of experiments, we dynamically adjust the exploration probability of $\epsilon$-greedy instead of fixing it. In particular, our set of experiments are based on: 
 
\begin{enumerate}
    

\item {\textbf{Using Fixed Exploration Probabilities.}} In this first set, for each $\epsilon$-greedy experiment, we choose $\epsilon$ values in the range $\epsilon=[0.0,1.0]$, in steps of 0.1.  In respect to softmax, we select the same range values for $\tau$ as $\epsilon$ but ignoring the 0.0 value because it causes a mathematical error. Instead of this value, we choose a value close to zero in order to replicate greedy behavior. In case of tie, we randomly select an action between all the tie actions. Additionally, in this first set of experiments, we consider a learning scenario where the adversary lets the agent learn a policy without perturbing any corrupted rewards and then it starts to craft attacks. In this case, the adversary starts to inject attacks from episode 1.000 to ensure that the learner has learned a policy at this episode.

\item \textbf{{Using Dynamic Exploration Probabilities.}} In this set of experiments, we dynamically adjust the exploration probability, $\epsilon$, of $\epsilon$-greedy instead of fixing it. The training starts with total exploration and ends with total exploitation. We use Equation~\ref{eq:rate} to update the exploration probability at each episode of the learning process. This $\epsilon$ value decreases as a linear function. 

\begin{equation}
    \epsilon_{t} =  1 - \left( \dfrac{e_{t}}{\cal{L}} \right)
    \label{eq:rate}
\end{equation}
where we obtain the dynamic exploration rate dividing the number of the current episode, $e_{t}$, by the number of total episodes of the training phase, $\cal{L}$. In the experiments, we set $\cal{L}$ to 5.000 episodes.
\end{enumerate}


\begin{figure*}[htbp!]
    \centering
    \includegraphics[width=0.9\textwidth]{fig1.png}
    \caption{Learning agent performance in an adversary free scenario.}
    \label{fig:baseBehave}
\end{figure*}


The adversary's goal is to produce the maximum deterioration in the learner policy performing the minimum number of attacks, so in the next section we study the results according to two criteria: the performance of the learning agent according to the Equation~\ref{eq:metric}, and the number of attacks injected. Finally, for all experiments $\alpha$ is fixed to 0.125 and $\gamma$ to 0.95, which are classical values in these kind of environments, so the results would be generalizeble to other settings.


\begin{figure*}[hbp!]
    \centering
    \includegraphics[width=0.9\textwidth]{cost1.png}
    \caption{Performance against cost. Configurations under an attack probability $p=0.3$ since episode 1000.}
    \label{fig:cost1}
\end{figure*}

\end{section}











\begin{section}{Results}



\label{sec:results}
Results in this section show the average of ten runs for each configuration and also, we use a slide window of a hundred episodes to reduce the variations between episodes. Following, we describe the results of each set of experiments.

\subsection{Learning without attacks}



First of all, in the first set of experiments, we show the performance of the learner agent without an enemy presence in Figure~\ref{fig:baseBehave}. 
Both plots show different learning curves each one corresponding to a different exploration rate. All configurations reach the goal before a thousand episodes. However, softmax takes more episodes for reaching their maximum values than $\epsilon$-greedy. 



\begin{subsection}{Using Fixed Exploration Probabilities}

In Figure~\ref{fig:baseBehave}, both exploration strategies show that the agent reaches the goal in fewer steps using low values of exploration. 
A higher rate of exploration performs more exploratory than exploitative actions and, although the agent still reaches the goal, the path is longer. On the contrary, low exploration rates lead the agent to perform more exploitative than exploratory actions, which cause to fall in local minimum from which the agent is not able to escape. Finally, Figure~\ref{fig:baseBehave} 
% Also, in the softmax strategy, we show that the agent which uses a greedy behavior (low temperature) falls in another local minimum but in this case, the learner is able to escape and finds the optimal path around episode 4000. This happens due to the softmax strategy lets the agent a small chance of exploring. Moreover, results demonstrate that we obtain the highest reward values using  exploration (temperature) values below 0.3.
also shows that the agent behaves almost randomly when $\epsilon$ or the temperature parameter are larger than 0.7.





Once we analyzed the performance using a \textit{free-of-attacks} scenario, we study how perturbations affect the system. In this set of experiments, the enemy injects adversarial rewards since episode 1000. At this time, a policy to reach one of the goals is learned by both exploration strategies.


In the learning processes of Figure~\ref{fig:attack1}, we fix the attack probability $p$ to $p=0.3$. In the case of $\epsilon$-greedy, Figure~\ref{fig:attack1} shows this attack probability is enough to destroy a total exploitation strategy policy. We observe that all the exploration rates obtain their worst results a few episodes after the emergence of corrupted rewards. After that, the learning processes associated to low exploration rates are able to improve their behaviors but without achieving the initial performances.

A similar analysis can be made with the softmax strategy at the right in Figure \ref{fig:attack1}. Few episodes later the irruption of corrupted rewards the performance of all configurations drops drastically. However, interestingly and in contrast to $\epsilon$-greedy, these agents are not able to slightly recover their performance. Figure~\ref{fig:attack1} demonstrates that low exploration rates are more resilient to small probabilities of attack. The reason is that these configurations allow the agent to reach a goal more frequently and, hence, it still receives more positives than negatives rewards, but it is unable to repair the policy.

%% FJGP: Recuerda que en la formalizacion de los ataques habiamos elegido la letra p para representar la probabilidad de los ataques. Es logico que ahora la utilicemos ahora tambien en la seccion de experimentos. Asi todo esta ligado y se entiende mejor.




Nevertheless, both strategies significantly decrease their performance under corrupted rewards, even with a low attack rate. When the adversary increases this rate, the agent is completely lost with attack probabilities higher that than $p=0.4$. In these cases, all the learned policies are wrong. Agents can occasionally reach the goal cell using exploration rates higher than $0.7$, but it is not enough to repair their policies.

%% FJGP: A veces llamas adversary y otras enemy al atacante. Habria que llamarlo siempre de la misma manera.

One of the main objective of the adversary is to avoid being detected.  Its objective is to reduce the discounted reward of the learner and also the number of attacks. Each attack has a cost associated. In this problem, we measure the cost as the number of attacks performed by the adversary. In Figure \ref{fig:cost1}, we combine the performance of the agent learner in the vertical axis and the cost in the x-axis. In this way, Figure \ref{fig:cost1} shows the two metrics the adversary wants to optimize. The lower-left the better for the adversary and the other way around for the learner. We show an interesting pattern in the first attacks which are the most harmful to deteriorate the policy.
We confirm that low exploration rates need more attacks to reduce their performances and, also, that few attacks affect considerably the highest exploration rates.




When we increase the attack probability $p$, the performance of all configurations decrease. We show in Figure \ref{fig:cost2} the minimum attack probability which is able to worsen any behavior. Also, we analyze the number of attacks needed to corrupt their policies. However, exploration strategies with exploration rates higher than $0.6$ reach one of the goals randomly. For this reason, we only compare configurations which use the learned policy. Comparing the Figures \ref{fig:cost1} and \ref{fig:cost2}, we observe that the higher the attack rate, the fewer number of attacks needed to destroy the policy. In the Figure \ref{fig:cost1}, the adversary performs more than $1000$ attacks to destroy the policies with the highest exploration rates. On the other plot, the adversary needs only less than 400 attacks to break the configurations which follow a policy, this means that the second attack strategy reduces the number of attacks more than 50\%.
In addition, attacking at each learning episode obtains the lowest cost for all configurations. Here, the adversary is able to destroy all the policies using only half as much as the setting used in the Figure \ref{fig:cost2}. 



%% FJGP: Esto de aqui abajo no lo acabo de entender. Redactarlo mejor.
%The exploration strategy in softmax is larger than in $\epsilon$-greedy so softmax keeps reaching one of the goals of the problems. This explains why softmax performs a higher number of attacks.
Due to exploration strategies, softmax keeps reaching at least one of the problem goals.   This explains why softmax performs a higher number of attacks.

\begin{figure*}[htpb!]
  \centering
  \includegraphics[width=0.9\textwidth]{fig2.png}
  \caption{Performance of $\epsilon$-greedy and softmax under an attack probability $p=0.3$.}
  \label{fig:attack1}
\end{figure*}

\begin{figure*}[htpb!]
    \centering
    \includegraphics[width=0.9\textwidth]{cost2.png}
    \caption{Performance against cost. Configurations under an attack probability $p=0.7$.}
    \label{fig:cost2}
\end{figure*}

The cause of this reduction is that report negative rewards frequently produces changes in the Q-table. The update function changes the value of the optimal actions for one negative and the agent selects actions that keep him away from the goal. For this reason, we consider consecutive attacks as the main factor to create successfully attack strategies.

In the light of the findings, we can conclude that softmax needs more episodes to learn and it suffers from adversarial perturbations further than $\epsilon$-greedy.
In addition, it is required an exhaustive tuning of  its temperature parameter in order to reach an appropiatte performance.

\end{subsection}


\begin{subsection}{Using Dynamic Exploration Probabilities}

In the second set of experiments, we use a dynamic probability of exploration, starting with the maximum exploration rate (i.e., $\epsilon=1$) and ending with total exploitation (i.e., $\epsilon=0$). 
%% FJGP: Este de aqui abajo no lo entiendo muy bien. Redactar mejor.
%In this way, the agent finds a solution when it unknowns everything about the domain, and in future episodes the agent applies the best set of actions to reach the goal.
In this way, the agent totally explores until it learns a policy and, in future episodes, starts to follow its policy.

\begin{figure*}[htpb!]
    \centering
    \includegraphics[width=0.9\textwidth]{dynamic.png}
    \caption{$\epsilon$-greedy using dynamic exploration. At the left, performing attacks from the beginning. And at the right, injecting attacks after episode 1000.}
    \label{fig:dynamic}
\end{figure*}

In Figure \ref{fig:dynamic}, we show the performance during the training updating the exploration rate every episode. In this set of experiments, unlike the former set, each line uses a different attack probability $p$.  We increase this probability from $p=0.0$ to $p=1.0$ in steps of $0.1$. The adversary return corrupted rewards from the beginning of the task. We use the configuration without adversary presence as reference to compare with the rest of experiments. In contrast to the first set of experiments, the agent takes more time  to achieve its maximum performance. In particular, the agent obtains the best result at the end of the training. Focusing on the behavior of the corrupted rewards, we observe that the learner is able to reach the goal in few steps more than using the optimal policy with attack probabilities lower than $p=0.3$.  Using attack probabilities of $p=0.3$ or $p=0.4$ produces that the policies get worsen when the agent reduces significantly its exploration rate. In addition, attack probabilities higher of  $p=0.5$ or higher produce that the agent gets lost in all episodes.

Figure \ref{fig:dynamic} demonstrate that the dynamic configuration is more robust to corrupted rewards than the fixed. In the fixed exploration experiments, we visualize huge variations between episodes when the agent is under enemy perturbations. On the contrary, the configuration with our adaptive exploration strategy achieves an almost identical behavior when the attack rate is low. Additionally, in terms of cost, the adversary needs to increase significantly the number of perturbations. This occurs because the attacked agent is still reaching the goal and the adversary continues crafting corrupted rewards. In this way, the adversary needs a greater number of attacks to destroy the learned policy and the agent may detect the insider.

\end{subsection}


\end{section}


\begin{section}{Conclusion}


%%No se si se me olvida algo. Probablemente, de una forma u otra, esto este contado, pero creo, como digo, que debería ser mas evidente poniendo items o algo asi como en: https://arxiv.org/ftp/arxiv/papers/1402/1402.0560.pdf. Ademas, las conclusiones Ruben no es un resumen del paper, sino contar lo que has aprendido escribiendolo. Esto me lo dijeron Fernando y Daniel un millon de veces antes de que me entrara en la cabeza :-) :-)

In this paper, we execute an exhaustive evaluation to test the robustness of different exploration/exploitation strategies. We use two set of experiments: fixed and dynamic exploration. The former modifies the probability of exploration and we compare these configurations under different adversary attack rates. In the latter, we use a dynamic exploration strategy to analyze the effects under different enemy strategies. Both set of experiments show that the highest probabilities of exploration are not able to reach the goal under the influence of an adversary, even with a low attack probability. Also, increasing the attack probability over the 40\%, the adversary destroys all the policies in a few steps.  As a result, we demonstrate that the first attacks after a learned policy and the consecutive adversary injections are the most harmful to deteriorate the agent behavior.

%% FJGP: Este parrafo de aqui no lo entiendo muy bien. Se habla de atacar en otras celdas que no sean metas, pero realmente no hemos incluido esos experimentos en el paper, luego yo lo quitaria...
%In our approach, attacking the reward requires to flip the sign of the value because we only report corrupted rewards when the agent reaches the goal. In other cases, the policy will not be affected because the agent receives correct rewards throughout training. Only when the adversary attacks all the time from the beginning and replaces the reward with a zero value, the learner identifies the goal as any other cell. Also, when the adversary injects attacks in every episode, the learner is not able to learn a policy to drive it to the goal. Then, we ratify the No Free Lunch Theorem. Additionally, we realize that by attacking only at the goal rewards, the adversary manages to affect the learner policy in every cell of the maze. 

From the adversary point of view, it aims to avoid being detected, so we propose a method to inject adversarial rewards reducing the number of attacks and the hardness of craft. In terms of cost, the adversary obtains its best performance attacking in each episode. This strategy produces policy failures performing less than 400 corrupted rewards in all the configurations. To decrease the number of attacks, we propose to attack only when the learner reaches the goal state. Also, we modify the attack probability to identify the best configuration for the adversary. Attacking  at  each learning episode is the cheapest adversary strategy. 

The experiments presented in this paper demonstrate some characteristics about exploration strategies in the presence of Adversarial perturbations.

\begin{enumerate}
    \item $\epsilon$-greedy is more robust than softmax. The probability of performing actions that differ from the policy learned in addition to the complexity of finding the right temperature parameters cause softmax obtains worse results than $\epsilon$-greedy. 
    \item Low exploration ratios are more resilient to attacks than larger values. Larger values of exploration produce a significant number of random actions aiming to reach any goal, regardless the number of steps or even the reward the agent receives.
    \item Dynamic exploration strategy is more resistant to perturbations. The dynamic exploration agent is able to resist  perturbation rates below 0.3 even from the beginning of the training and it obtains a performance similar to the free adversary scenario. We consider that this strategy is an excellent alternative to train an agent when there are chances of obtaining disturbances in the reward function.
    \item As opposed to what one might think, performing attacks every episode becomes the cheapest option. Performing consecutive attacks are more harmful because the agent can not adjust their Q table values to correct these perturbations in future episodes.
\end{enumerate}

%% FJGP: Yo creo que deberiamos hacer mas evidente que:
%% 1. e-greedy es mas robusta frente a ataques que softmax, la dificultad de encontrar los parametros de temperatura
%% 2. que bajos niveles de exploracion son mas resistentes a ataques que niveles altos, los ataques mas dañinos ....
%% 3. que la forma mas robusta de exploracion frente a ataques es la exploración dinamica, puesto que con probabilidades de ataque por debajo de XX, es capaz de alcanzar niveles de rendimiento similares a cuando no se ataca
%% 4. que, al contrario de lo que se podria pensar, atacar en cada episodio acaba siendo lo menos costoso. 


% Nevertheless, with low attack probabilities, the dynamic exploration agent is able to resist these perturbations even from the beginning of the training and it obtains a similar performance to an agent without the adversary presence.


%% FJGP: Si no lo hemos demostrado experimentalmente, yo quitaria esta afirmacion
%We managed  to significantly reduce the amount of attacks compared to other state-of-art approaches which attack the reward function.

In future work, we would generalize our approach using other domains such as the Atari games used in Deep RL. Also, we would analyze the behavior of other agent configurations. Testing more elaborate attacks strategies such as introducing intermediate rewards or developing smart adversary behaviors are some examples of an exciting topic for future research.

\end{section}

\begin{section}{Acknowledgment}

This research is granted by Repsol, the Spanish Goverment (Ministerio de Economia y Empresa) and FEDER, UE funds under project TIN2017-88476-C2-2-R.
This research was funded in part by JPMorgan Chase \& Co. Any views or opinions expressed herein are solely those of the authors listed, and may differ from the views and opinions expressed by JPMorgan Chase \& Co. or its affiliates. This material is not a product of the Research Department of J.P. Morgan Securities LLC. This material should not be construed as an individual recommendation for any particular client and is not intended as a recommendation of particular securities, financial instruments or strategies for a particular client. This material does not constitute a solicitation or offer in any jurisdiction.

\end{section}

%\newpage
%
% ---- Bibliography ----
%
% BibTeX users should specify bibliography style 'splncs04'.
% References will then be sorted and formatted in the correct style.
%
\bibliographystyle{ecai}
\bibliography{references}
%

\end{document}

main.bbl
\begin{thebibliography}{10}

\bibitem{behzadan2017vulnerability}
Vahid Behzadan and Arslan Munir, `Vulnerability of deep reinforcement learning
  to policy induction attacks', in {\em International Conference on Machine
  Learning and Data Mining in Pattern Recognition}, pp. 262--275. Springer,
  (2017).

\bibitem{tong2019}
Tong Chen, Jiqiang Liu, Yingxiao Xiang, Wenjia Niu, Endong Tong, and Zhen Han,
  `Adversarial attack and defense in reinforcement learning-from ai security
  view', {\em Cybersecurity}, {\bf 2}, (12 2019).

\bibitem{Everitt2017rc}
Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg,
  `Reinforcement learning with a corrupted reward signal', in {\em Proceedings
  of the Twenty-Sixth International Joint Conference on Artificial
  Intelligence, {IJCAI} 2017, Melbourne, Australia, August 19-26, 2017}, pp.
  4705--4713, (2017).

\bibitem{ferdowsi2018robust}
Aidin Ferdowsi, Ursula Challita, Walid Saad, and Narayan~B Mandayam, `Robust
  deep reinforcement learning for security and safety in autonomous vehicle
  systems', in {\em 2018 21st International Conference on Intelligent
  Transportation Systems (ITSC)}, pp. 307--312. IEEE, (2018).

\bibitem{DBLP:journals/corr/HuangPGDA17}
Sandy~H. Huang, Nicolas Papernot, Ian~J. Goodfellow, Yan Duan, and Pieter
  Abbeel, `Adversarial attacks on neural network policies', {\em CoRR}, {\bf
  abs/1702.02284}, (2017).

\bibitem{kos2017delving}
Jernej Kos and Dawn Song, `Delving into adversarial attacks on deep policies',
  {\em arXiv preprint arXiv:1705.06452}, (2017).

\bibitem{kuderer2015learning}
Markus Kuderer, Shilpa Gulati, and Wolfram Burgard, `Learning driving styles
  for autonomous vehicles from demonstration', in {\em 2015 IEEE International
  Conference on Robotics and Automation (ICRA)}, pp. 2641--2646. IEEE, (2015).

\bibitem{DBLP:journals/jmlr/LevineFDA16}
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel, `End-to-end
  training of deep visuomotor policies', {\em J. Mach. Learn. Res.}, {\bf 17},
  39:1--39:40, (2016).

\bibitem{liu2019data}
Fang Liu and Ness Shroff, `Data poisoning attacks on stochastic bandits', in
  {\em International Conference on Machine Learning}, pp. 4042--4050, (2019).

\bibitem{ma2019policy}
Yuzhe Ma, Xuezhou Zhang, Wen Sun, and Xiaojin Zhu, `Policy poisoning in batch
  reinforcement learning and control', in {\em Advances in Neural Information
  Processing Systems}, (2019).

\bibitem{mnih2013playing}
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis
  Antonoglou, Daan Wierstra, and Martin Riedmiller, `Playing atari with deep
  reinforcement learning', {\em arXiv preprint arXiv:1312.5602}, (2013).

\bibitem{morere18a}
Philippe Morere and Fabio Ramos, `Bayesian rl for goal-only rewards', in {\em
  Proceedings of The 2nd Conference on Robot Learning}, eds., Aude Billard,
  Anca Dragan, Jan Peters, and Jun Morimoto, volume~87 of {\em Proceedings of
  Machine Learning Research}, pp. 386--398. PMLR, (29--31 Oct 2018).

\bibitem{pattanaik2018robust}
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish
  Chowdhary, `Robust deep reinforcement learning with adversarial attacks', in
  {\em Proceedings of the 17th International Conference on Autonomous Agents
  and MultiAgent Systems}, pp. 2040--2042. International Foundation for
  Autonomous Agents and Multiagent Systems, (2018).

\bibitem{pinto2017robust}
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta, `Robust
  adversarial reinforcement learning', {\em arXiv preprint arXiv:1703.02702},
  (2017).

\bibitem{reinke2017}
Chris Reinke, Eiji Uchibe, and Kenji Doya, `Average reward optimization with
  multiple discounting reinforcement learners', in {\em Neural Information
  Processing}, eds., Derong Liu, Shengli Xie, Yuanqing Li, Dongbin Zhao, and
  El-Sayed~M. El-Alfy, pp. 789--800, Cham, (2017). Springer International
  Publishing.

\bibitem{romoff2018reward}
Joshua Romoff, Alexandre Pich{\'e}, Peter Henderson, Vincent Francois-Lavet,
  and Joelle Pineau, `Reward estimation for variance reduction in deep
  reinforcement learning', {\em arXiv preprint arXiv:1805.03359}, (2018).

\bibitem{roy2017reinforcement}
Aurko Roy, Huan Xu, and Sebastian Pokutta, `Reinforcement learning under model
  mismatch', in {\em Advances in Neural Information Processing Systems}, pp.
  3043--3052, (2017).

\bibitem{DBLP:journals/corr/SzegedyZSBEGF13}
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan,
  Ian Goodfellow, and Rob Fergus, `Intriguing properties of neural networks',
  in {\em International Conference on Learning Representations}, (2014).

\bibitem{alphastarblog2}
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg,
  Wojciech~M. Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard
  Powell, Timo Ewalds, Dan Horgan, Manuel Kroiss, Ivo Danihelka, John Agapiou,
  Junhyuk Oh, Valentin Dalibard, David Choi, Laurent Sifre, Yury Sulsky, Sasha
  Vezhnevets, James Molloy, Trevor Cai, David Budden, Tom Paine, Caglar
  Gulcehre, Ziyu Wang, Tobias Pfaff, Toby Pohlen, Yuhuai Wu, Dani Yogatama,
  Julia Cohen, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap,
  Chris Apps, Koray Kavukcuoglu, Demis Hassabis, and David Silver.
\newblock {AlphaStar: Mastering the Real-Time Strategy Game StarCraft II},
  2019.

\bibitem{DBLP:journals/corr/abs-1810-01032}
Jingkang Wang, Yang Liu, and Bo~Li, `Reinforcement learning with perturbed
  rewards', {\em CoRR}, {\bf abs/1810.01032}, (2018).

\end{thebibliography}

references.bib
@article{mnih2013playing,
  title={Playing atari with deep reinforcement learning},
  author={Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Graves, Alex and Antonoglou, Ioannis and Wierstra, Daan and Riedmiller, Martin},
  journal={arXiv preprint arXiv:1312.5602},
  year={2013}
}

@article{silver2016mastering,
  title={Mastering the game of Go with deep neural networks and tree search},
  author={Silver, David and Huang, Aja and Maddison, Chris J and Guez, Arthur and Sifre, Laurent and Van Den Driessche, George and Schrittwieser, Julian and Antonoglou, Ioannis and Panneershelvam, Veda and Lanctot, Marc and others},
  journal={nature},
  volume={529},
  number={7587},
  pages={484},
  year={2016},
  publisher={Nature Publishing Group}
}

@inproceedings{goodfellow2014generative,
  title={Generative adversarial nets},
  author={Goodfellow, Ian and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and Warde-Farley, David and Ozair, Sherjil and Courville, Aaron and Bengio, Yoshua},
  booktitle={Advances in neural information processing systems},
  pages={2672--2680},
  year={2014}
}


@inproceedings{DBLP:journals/corr/SzegedyZSBEGF13,
title	= {Intriguing properties of neural networks},
author	= {Christian Szegedy and Wojciech Zaremba and Ilya Sutskever and Joan Bruna and Dumitru Erhan and Ian Goodfellow and Rob Fergus},
year	= {2014},
URL	= {http://arxiv.org/abs/1312.6199},
booktitle	= {International Conference on Learning Representations}
}




@misc{alphastarblog,
  title="{AlphaStar: Mastering the Real-Time Strategy Game StarCraft II}",
  author={Vinyals, Oriol and Babuschkin, Igor and Chung, Junyoung and Mathieu, Michael and Jaderberg, Max and Czarnecki, Wojciech M. and Dudzik, Andrew and Huang, Aja and Georgiev, Petko and Powell, Richard and Ewalds, Timo and Horgan, Dan and Kroiss, Manuel and Danihelka, Ivo and Agapiou, John and Oh, Junhyuk and Dalibard, Valentin and Choi, David and Sifre, Laurent and Sulsky, Yury and Vezhnevets, Sasha and Molloy, James and Cai, Trevor and Budden, David and Paine, Tom and Gulcehre, Caglar and Wang, Ziyu and Pfaff, Tobias and Pohlen, Toby and Wu, Yuhuai and Yogatama, Dani and Cohen, Julia and McKinney, Katrina and Smith, Oliver and Schaul, Tom and Lillicrap, Timothy and Apps, Chris and Kavukcuoglu, Koray and Hassabis, Demis and Silver, David},
  howpublished={\url{https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/}},
  year={2019}
}

@inproceedings{havens2018online,
  title={Online robust policy learning in the presence of unknown adversaries},
  author={Havens, Aaron and Jiang, Zhanhong and Sarkar, Soumik},
  booktitle={Advances in Neural Information Processing Systems},
  pages={9916--9926},
  year={2018}
}


@misc{alphastarblog2,
  title="{AlphaStar: Mastering the Real-Time Strategy Game StarCraft II}",
  author={Vinyals, Oriol and Babuschkin, Igor and Chung, Junyoung and Mathieu, Michael and Jaderberg, Max and Czarnecki, Wojciech M. and Dudzik, Andrew and Huang, Aja and Georgiev, Petko and Powell, Richard and Ewalds, Timo and Horgan, Dan and Kroiss, Manuel and Danihelka, Ivo and Agapiou, John and Oh, Junhyuk and Dalibard, Valentin and Choi, David and Sifre, Laurent and Sulsky, Yury and Vezhnevets, Sasha and Molloy, James and Cai, Trevor and Budden, David and Paine, Tom and Gulcehre, Caglar and Wang, Ziyu and Pfaff, Tobias and Pohlen, Toby and Wu, Yuhuai and Yogatama, Dani and Cohen, Julia and McKinney, Katrina and Smith, Oliver and Schaul, Tom and Lillicrap, Timothy and Apps, Chris and Kavukcuoglu, Koray and Hassabis, Demis and Silver, David},
  year={2019}
}

@article{DBLP:journals/jmlr/LevineFDA16,
  author    = {Sergey Levine and
               Chelsea Finn and
               Trevor Darrell and
               Pieter Abbeel},
  title     = {End-to-End Training of Deep Visuomotor Policies},
  journal   = {J. Mach. Learn. Res.},
  volume    = {17},
  pages     = {39:1--39:40},
  year      = {2016},
  url       = {http://jmlr.org/papers/v17/15-522.html},
  timestamp = {Wed, 10 Jul 2019 15:28:13 +0200},
  biburl    = {https://dblp.org/rec/bib/journals/jmlr/LevineFDA16},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}


@inproceedings{ferdowsi2018robust,
  title={Robust deep reinforcement learning for security and safety in autonomous vehicle systems},
  author={Ferdowsi, Aidin and Challita, Ursula and Saad, Walid and Mandayam, Narayan B},
  booktitle={2018 21st International Conference on Intelligent Transportation Systems (ITSC)},
  pages={307--312},
  year={2018},
  organization={IEEE}
}

@inproceedings{kuderer2015learning,
  title={Learning driving styles for autonomous vehicles from demonstration},
  author={Kuderer, Markus and Gulati, Shilpa and Burgard, Wolfram},
  booktitle={2015 IEEE International Conference on Robotics and Automation (ICRA)},
  pages={2641--2646},
  year={2015},
  organization={IEEE}
}


@inproceedings{dalvi2004adversarial,
  title={Adversarial classification},
  author={Dalvi, Nilesh and Domingos, Pedro and Sanghai, Sumit and Verma, Deepak and others},
  booktitle={Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining},
  pages={99--108},
  year={2004},
  organization={ACM}
}

@inproceedings{lowd2005adversarial,
  title={Adversarial learning},
  author={Lowd, Daniel and Meek, Christopher},
  booktitle={Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining},
  pages={641--647},
  year={2005},
  organization={ACM}
}

@article{li2017adversarial,
  title={Adversarial learning for neural dialogue generation},
  author={Li, Jiwei and Monroe, Will and Shi, Tianlin and Jean, S{\'e}bastien and Ritter, Alan and Jurafsky, Dan},
  journal={arXiv preprint arXiv:1701.06547},
  year={2017}
}

@inproceedings{huang2011adversarial,
  title={Adversarial machine learning},
  author={Huang, Ling and Joseph, Anthony D and Nelson, Blaine and Rubinstein, Benjamin IP and Tygar, JD},
  booktitle={Proceedings of the 4th ACM workshop on Security and artificial intelligence},
  pages={43--58},
  year={2011},
  organization={ACM}
}

@article{radford2015unsupervised,
  title={Unsupervised representation learning with deep convolutional generative adversarial networks},
  author={Radford, Alec and Metz, Luke and Chintala, Soumith},
  journal={arXiv preprint arXiv:1511.06434},
  year={2015}
}

@article{kos2017delving,
  title={Delving into adversarial attacks on deep policies},
  author={Kos, Jernej and Song, Dawn},
  journal={arXiv preprint arXiv:1705.06452},
  year={2017}
}

@article{goodfellow2014explaining,
  title={Explaining and Harnessing Adversarial Examples},
  author={Goodfellow, Ian J and Shlens, Jonathon and Szegedy, Christian},
  journal={arXiv preprint arXiv:1412.6572},
  year={2014}
}

@inproceedings{pattanaik2018robust,
  title={Robust deep reinforcement learning with adversarial attacks},
  author={Pattanaik, Anay and Tang, Zhenyi and Liu, Shuijing and Bommannan, Gautham and Chowdhary, Girish},
  booktitle={Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems},
  pages={2040--2042},
  year={2018},
  organization={International Foundation for Autonomous Agents and Multiagent Systems}
}


@article{mnih2015human,
  title={Human-level control through deep reinforcement learning},
  author={Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Rusu, Andrei A and Veness, Joel and Bellemare, Marc G and Graves, Alex and Riedmiller, Martin and Fidjeland, Andreas K and Ostrovski, Georg and others},
  journal={Nature},
  volume={518},
  number={7540},
  pages={529},
  year={2015},
  publisher={Nature Publishing Group}
}


@inproceedings{behzadan2017vulnerability,
  title={Vulnerability of deep reinforcement learning to policy induction attacks},
  author={Behzadan, Vahid and Munir, Arslan},
  booktitle={International Conference on Machine Learning and Data Mining in Pattern Recognition},
  pages={262--275},
  year={2017},
  organization={Springer}
}

@article{pinto2017robust,
  title={Robust adversarial reinforcement learning},
  author={Pinto, Lerrel and Davidson, James and Sukthankar, Rahul and Gupta, Abhinav},
  journal={arXiv preprint arXiv:1703.02702},
  year={2017}
}

@misc{openAI16,
  Author = {Greg Brockman and Vicki Cheung and Ludwig Pettersson and Jonas Schneider and John Schulman and Jie Tang and Wojciech Zaremba},
  Title = {OpenAI Gym},
  Year = {2016},
  Eprint = {arXiv:1606.01540},
}

@article{lin2017tactics,
  title={Tactics of adversarial attack on deep reinforcement learning agents},
  author={Lin, Yen-Chen and Hong, Zhang-Wei and Liao, Yuan-Hong and Shih, Meng-Li and Liu, Ming-Yu and Sun, Min},
  journal={arXiv preprint arXiv:1703.06748},
  year={2017}
}

@inproceedings{carlini2017towards,
  title={Towards evaluating the robustness of neural networks},
  author={Carlini, Nicholas and Wagner, David},
  booktitle={2017 IEEE Symposium on Security and Privacy (SP)},
  pages={39--57},
  year={2017},
  organization={IEEE}
}

@book{rubinstein2013cross,
  title={The Cross-Entropy Method: A Unified Approach to Combinatorial Optimization, Monte-Carlo Simulation and Machine Learning},
  author={Rubinstein, Reuven Y and Kroese, Dirk P},
  year={2013},
  publisher={Springer Science \& Business Media}
}

@article{co2018procedural,
  title={Procedural Noise Adversarial Examples for Black-Box Attacks on Deep Neural Networks},
  author={Co, Kenneth T and Mu{\~n}oz-Gonz{\'a}lez, Luis and Lupu, Emil C},
  journal={arXiv preprint arXiv:1810.00470},
  year={2018}
}

@article{DBLP:journals/corr/HuangPGDA17,
  author    = {Sandy H. Huang and
               Nicolas Papernot and
               Ian J. Goodfellow and
               Yan Duan and
               Pieter Abbeel},
  title     = {Adversarial Attacks on Neural Network Policies},
  journal   = {CoRR},
  volume    = {abs/1702.02284},
  year      = {2017},
  url       = {http://arxiv.org/abs/1702.02284},
  archivePrefix = {arXiv},
  eprint    = {1702.02284},
  timestamp = {Mon, 13 Aug 2018 16:48:47 +0200},
  biburl    = {https://dblp.org/rec/bib/journals/corr/HuangPGDA17},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}



@inproceedings{barreno2006can,
  title={Can machine learning be secure?},
  author={Barreno, Marco and Nelson, Blaine and Sears, Russell and Joseph, Anthony D and Tygar, J Doug},
  booktitle={Proceedings of the 2006 ACM Symposium on Information, computer and communications security},
  pages={16--25},
  year={2006},
  organization={ACM}
}

@article{DBLP:journals/corr/abs-1810-01032,
  author    = {Jingkang Wang and
               Yang Liu and
               Bo Li},
  title     = {Reinforcement Learning with Perturbed Rewards},
  journal   = {CoRR},
  volume    = {abs/1810.01032},
  year      = {2018},
  url       = {http://arxiv.org/abs/1810.01032},
  archivePrefix = {arXiv},
  eprint    = {1810.01032},
  timestamp = {Tue, 30 Oct 2018 10:49:09 +0100},
  biburl    = {https://dblp.org/rec/bib/journals/corr/abs-1810-01032},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

@article{DBLP:journals/corr/abs-1710-00814,
  author    = {Yen{-}Chen Lin and
               Ming{-}Yu Liu and
               Min Sun and
               Jia{-}Bin Huang},
  title     = {Detecting Adversarial Attacks on Neural Network Policies with Visual
               Foresight},
  journal   = {CoRR},
  volume    = {abs/1710.00814},
  year      = {2017},
  url       = {http://arxiv.org/abs/1710.00814},
  archivePrefix = {arXiv},
  eprint    = {1710.00814},
  timestamp = {Mon, 13 Aug 2018 16:48:49 +0200},
  biburl    = {https://dblp.org/rec/bib/journals/corr/abs-1710-00814},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

@article{amodei2016concrete,
  title={Concrete problems in AI safety},
  author={Amodei, Dario and Olah, Chris and Steinhardt, Jacob and Christiano, Paul and Schulman, John and Man{\'e}, Dan},
  journal={arXiv preprint arXiv:1606.06565},
  year={2016}
}

@inproceedings{krueger2016active,
  title={Active reinforcement learning: Observing rewards at a cost},
  author={Krueger, David and Leike, Jan and Evans, Owain and Salvatier, John}
}

@article{romoff2018reward,
  title={Reward estimation for variance reduction in deep reinforcement learning},
  author={Romoff, Joshua and Pich{\'e}, Alexandre and Henderson, Peter and Francois-Lavet, Vincent and Pineau, Joelle},
  journal={arXiv preprint arXiv:1805.03359},
  year={2018}
}

@inproceedings{roy2017reinforcement,
  title={Reinforcement learning under model mismatch},
  author={Roy, Aurko and Xu, Huan and Pokutta, Sebastian},
  booktitle={Advances in Neural Information Processing Systems},
  pages={3043--3052},
  year={2017}
}

@inproceedings{ma2019policy,
  title={Policy Poisoning in Batch Reinforcement Learning and Control},
  author={Ma, Yuzhe and Zhang, Xuezhou and Sun, Wen and Zhu, Xiaojin},
  booktitle={Advances in Neural Information Processing Systems},
  year={2019}
}

@inproceedings{liu2019data,
  title={Data Poisoning Attacks on Stochastic Bandits},
  author={Liu, Fang and Shroff, Ness},
  booktitle={International Conference on Machine Learning},
  pages={4042--4050},
  year={2019}
}



@inproceedings{Everitt2017rc,
  author = {Everitt, Tom and Krakovna, Victoria and Orseau, Laurent and Hutter, Marcus and Legg, Shane},
  title = {Reinforcement Learning with a Corrupted Reward Signal},
  booktitle = {Proceedings of the Twenty-Sixth International Joint Conference on
               Artificial Intelligence, {IJCAI} 2017, Melbourne, Australia, August
               19-26, 2017},
  pages = {4705--4713},
  year = {2017},
  doi = {10.24963/ijcai.2017/656},
  url = {	https://doi.org/10.24963/ijcai.2017/656},
  abstract = {No real-world reward function is perfect. Sensory errors and software bugs may result in agents getting higher (or lower) rewards than they should. For example, a reinforcement learning agent may prefer states where a sensory error gives it the maximum reward, but where the true reward is actually small. We formalise this problem as a generalised Markov Decision Problem called Corrupt Reward MDP. Traditional RL methods fare poorly in CRMDPs, even under strong simplifying assumptions and when trying to compensate for the possibly corrupt rewards. Two ways around the problem are investigated. First, by giving the agent richer data, such as in inverse reinforcement learning and semi-supervised reinforcement learning, reward corruption stemming from systematic sensory errors may sometimes be completely managed. Second, by using randomisation to blunt the agent's optimisation, reward corruption can be partially managed under some assumptions.},
  archiveprefix = {arXiv},
  eprint = {1705.08417},
  primaryclass = {cs.AI},
  blog = {http://www.tomeveritt.se/paper/2017/05/29/reinforcement-learning-with-corrupted-reward-channel.html}
}

@article{tong2019,
author = {Chen, Tong and Liu, Jiqiang and Xiang, Yingxiao and Niu, Wenjia and Tong, Endong and Han, Zhen},
year = {2019},
month = {12},
pages = {},
title = {Adversarial attack and defense in reinforcement learning-from AI security view},
volume = {2},
journal = {Cybersecurity}
}

@article{xiao2018security,
  title={Security in mobile edge caching with reinforcement learning},
  author={Xiao, Liang and Wan, Xiaoyue and Dai, Canhuang and Du, Xiaojiang and Chen, Xiang and Guizani, Mohsen},
  journal={IEEE Wireless Communications},
  volume={25},
  number={3},
  pages={116--122},
  year={2018},
  publisher={IEEE}
}

@article{DBLP:journals/tec/OnePixel,
  author    = {Jiawei Su and
               Danilo Vasconcellos Vargas and
               Kouichi Sakurai},
  title     = {One Pixel Attack for Fooling Deep Neural Networks},
  journal   = {{IEEE} Trans. Evolutionary Computation},
  volume    = {23},
  number    = {5},
  pages     = {828--841},
  year      = {2019},
  url       = {https://doi.org/10.1109/TEVC.2019.2890858},
  doi       = {10.1109/TEVC.2019.2890858},
  timestamp = {Fri, 11 Oct 2019 15:34:19 +0200},
  biburl    = {https://dblp.org/rec/bib/journals/tec/SuVS19},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

@InProceedings{morere18a,
  title = 	 {Bayesian RL for Goal-Only Rewards},
  author = 	 {Morere, Philippe and Ramos, Fabio},
  booktitle = 	 {Proceedings of The 2nd Conference on Robot Learning},
  pages = 	 {386--398},
  year = 	 {2018},
  editor = 	 {Billard, Aude and Dragan, Anca and Peters, Jan and Morimoto, Jun},
  volume = 	 {87},
  series = 	 {Proceedings of Machine Learning Research},
  address = 	 {},
  month = 	 {29--31 Oct},
  publisher = 	 {PMLR}
}

@InProceedings{reinke2017,
author="Reinke, Chris and Uchibe, Eiji and Doya, Kenji", 
editor="Liu, Derong and Xie, Shengli
and Li, Yuanqing and Zhao, Dongbin and El-Alfy, El-Sayed M.",
title="Average Reward Optimization with Multiple Discounting Reinforcement Learners",
booktitle="Neural Information Processing",
year="2017",
publisher="Springer International Publishing",
address="Cham",
pages="789--800"
}

