\section{Introduction}

Combining deep neural networks with reinforcement learning has produced impressive results on challenging problems, such as playing Chess \cite{silver2018general}, Atari games \cite{mnih2015human}, and robotics tasks \cite{schulman2017proximal}. However, results have bifurcated between two competing approaches: Q-learning-based approaches that require learning an (action conditioned) value estimate and actor-critic policy gradient (AC-PG, or just PG)\footnote{For simplicity, we refer to actor-critic policy gradient as policy gradient, even though some early policy gradient approaches, such as REINFORCE \cite{williams1992simple}, do not learn a value function.} approaches that learn both a policy and value estimate. PG offers many theoretical advantages over Q-learning, such as natural support for continuous control problems and the ability to learn stochastic policies. However, until recently, PG approaches \cite{mnih2016asynchronous, schulman2017proximal, cobbe2021phasic}, while strong on continuous control problems, have under-performed Q-learning approaches on complex vision-based problems \cite{mnih2015human, hessel2018rainbow, badia2020agent57}, restricting the use of PG in this domain.

This paper aims to close the gap between PG and Q-learning methods by showing that the common practice of jointly learning value and policy with a shared representation negatively affects the algorithm's performance.\footnote{PPO \cite{schulman2017proximal}, A3C \cite{mnih2016asynchronous}, and Muslie \cite{hessel2021muesli} all use a joint network when training on vision-based discrete action environments. PPG \cite{cobbe2021phasic} uses a dual network with an auxiliary task, which we discuss in more detail in Section \ref{sec:background}} We also demonstrate an order-of-magnitude difference in noise levels between these two tasks and argue that this makes these two tasks poorly aligned.

In light of this result, we introduce a new algorithm, based on Proximal Policy Optimization (PPO) \cite{schulman2017proximal}, called Dual Network Architecture (DNA). We test DNA empirically on a subset of the Arcade Learning Environment (ALE) \cite{bellemare2013arcade} called Atari-5.\footnote{This is a parallel submission. An anonymous copy is provided in the supplementary materials.},  Our results show a strong increase in performance compared to both PPO and another dual network model Phasic Policy Gradient (PPG) \cite{cobbe2021phasic}. Our model also outperforms Rainbow DQN \cite{hessel2018rainbow} on four of the five games tested, even under more challenging environmental settings. 

We summarize our contributions. First, we provide empirical results showing an order-of-magnitude difference in the noise scale between policy gradient and value learning. Second, we give evidence for the benefit of using low \textit{bias} value estimates for value learning, but low \textit{variance} estimates for advantage estimates. Finally, we introduce and justify our dual network constrained distillation algorithm DNA with empirical results on ALE.
 
\section{Preliminaries and Related Work}
\label{sec:background}

\paragraph{Multi-task learning} Multi-task learning is the process of learning a shared representation that can be used to solve multiple tasks in the hope that \textit{constructive interference} between the tasks will lead to better performance \cite{ruder2017overview}. However, this is not always the case. When tasks are unaligned \textit{destructive interference}, can instead reduce the performance of the model \cite{Wu2020Understanding}. We argue that large differences between the noise scale in value learning and policy gradient make these tasks poorly unaligned.

\paragraph{TD($\lambda$) Return Estimation}


Policy gradient algorithms often make use of a value estimate as a baseline \cite{mnih2016asynchronous}, as well as for estimating the value of truncated trajectories \cite{cichosz1994truncating}. To better facilitate control over the noise levels for policy and value learning, our work makes use of two different return estimations, both using TD($\lambda$) \cite{sutton1988learning}. Given value estimates $V(\cdot)$ from the value network we define the n-step value estimate for some state $s_t$ taken at time $t$, and their exponentially weighted sum as

\begin{align}
    \text{NSTEP}^{(\gamma, k)}(s_t) &:= \sum_{i=0}^{k-1} \gamma^i r_{t+i} + \gamma^k V(s_{t+k}),\\
    \text{TD}^{(\gamma, \lambda)}(s_t) &:= (1-\lambda) \sum_{k=1}^{\infty} \lambda^{k-1} \text{NSTEP}^{(\gamma, k)}(s_t).
\end{align}

For values of $\lambda$ close to $1$, more weight is assigned to longer n-step return estimates, and less to shorter ones. There has been a long-standing belief that shorter n-step returns generate more biased estimates, whereas longer n-step estimates, due to summing over many stochastic rewards, have higher variance.\footnote{While there are cases (i.e. where rewards have negative covariance) where this is not the case, empirical experiments confirm the intuition that longer n-step estimates generally have higher variance and lower bias.} For a more thorough discussion on this topic see \cite{kearns2000bias}.


\paragraph{Noise Scale.} Noise scale is a measure of `signal to noise' in stochastic gradient descent (SGD). While SGD provides unbiased gradient estimates, these estimates are typically very noisy. Each sampled gradient estimate $\hat{G}$ can be thought of as a sum of the true gradient $\vec{G}$ and some noise vector $\vec{\sigma}$. A method for estimating the ratio of the magnitude of this implied noise vector to the magnitude of the true gradient was proposed by \cite{mccandlish2018empirical} who show that their efficient-to-calculate \textit{simple noise scale} is a good match for the noise scale. They also show that the noise scale provides useful information about the choice of mini-batch size to use when estimating gradients.

\paragraph{Phasic Policy Gradient.} Most similar to our work is Phasic Policy Gradient (PPG) \cite{cobbe2021phasic}. Like DNA, PPG has three distinct phases during training and uses two independent networks. However, there are several important distinctions in our work. First, we forgo the large replay buffer, relying instead on learning entirely from recent experience, significantly reducing the memory requirements of our algorithm. Second, we make use of a distillation phase rather than an auxiliary phase.\footnote{PPG refer to their third phase as a distillation phase, however, because the update trains both the policy and value networks on value estimates generated from rollouts, this is better thought of as an auxiliary task. We discuss this in more detail in Section \ref{sec:distillation}} Finally, our work reduces gradient noise by using two different return estimators calibrated for the properties of each task. We assess the impact of these differences on the performance of the agent in Section \ref{sec:results}.

\section{The Noise Properties of Value Learning and Policy Gradient}
\label{sec:noise}

Here we examine the noise properties of the loss functions for policy gradient and value learning and show that policy gradient has a much higher noise level than value learning. This implies different challenges in the optimization problem. Specifically, that policy should be learned with a larger mini-batch size than value learning and that return estimates should be adapted appropriately.

\subsection{Motivating Example}

To motivate our investigation into noise levels, we consider the task of learning two independent functions $F_1(x) := \sin(5x)+\text{N}(0, \sigma_1^2)$ and $F_2(x) := \cos(5x)+\text{N}(0, \sigma_2^2)$, where $\text{N}(\mu,\sigma^2)$ is Gaussian noise with mean $\mu$ and standard deviation $\sigma$,  over the domain $[-\pi, \pi]$. We fix $\sigma_2$ = 1, and vary $\sigma_1$ on a log scale from $0.1$ to $100$. We trained two 3-layer multi-layer perceptions (MLPs) on this problem.\footnote{It is worth noting that we did not find the same result when using shallow networks. That is, dual and joint models performed similarly when using a 2-layer MLP.}  The first was a joint model, using a first hidden layer of 1024 units and a second hidden layer of 2048 units, followed by two output heads. Our second model was a dual network consisting of two independent MLPs with 1024 units on each hidden layer.\footnote{This number of hidden units was chosen so that both models had the same number of parameters.} All models used ReLU activations in between linear layers, and error was measured as mean-squared-error (MSE) between the predicted value and the \textit{noise free} true value. The results are presented in Figure \ref{fig:toy_noise}. At low noise levels, the tasks did not interfere, but as the noise of the first task increased, performance on the second task eventually degraded for the joint network, but not the dual network. While simple, this experiment demonstrates the impact one noisy task can have on another when learned jointly by a deep neural network.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.49\textwidth]{figures/toy_f1.pdf}
    \includegraphics[width=0.49\textwidth]{figures/toy_f2.pdf}
    \caption{A depiction of the problem of destructive interference on a toy problem. $L_{T1}$ and $L_{T2}$ refer to the mean-squared-error on the first and second task respectively.%
   
    Shading indicates standard error over 100 seeds.}
    \label{fig:toy_noise}
\end{figure}

\subsection{Noise Scale in Reinforcement Learning}


To assess the differences in noise levels in our experiments, we measure the \textit{gradient noise scale} of the policy gradient and value loss gradient. We did this by learning an estimate of the noise scale, developed by \cite{mccandlish2018empirical} called \textit{simple noise scale} defined as
\begin{align}
    \bs := \frac{\text{tr}(\Sigma)}{|\vec{G}|^2},
\end{align}
where $\Sigma$ is the gradient covariant matrix, and $\vec{G}$ is the true gradient if the entire batch was used. For convenience we also use the notation $\sigma := \sqrt{\bs}$, which can interpreted as the ratio of the length of the implied noise vector to the length of the true gradient vector. That is, a kind of noise-to-signal ratio. As \cite{mccandlish2018empirical} have shown, an unbiased estimate of \bs can be found efficiently by generating gradient estimates \gbs using a small mini-batch of size \bbs as well as the gradient \gbl using a large mini-batch of size \bbb, then calculating

\begin{align}
     |\mathcal{G}|^2 &:= \frac{1}{\bbb - \bbs}\large(  \bbb |\gbl|^2 - \bbs |\gbs|^2   \large) \\
     \mathcal{S} &:= \frac{1}{1/\bbs - 1/ \bbb} \large( |\gbs|^2 - |\gbl|^2   \large),
\end{align}

where $\mathbb{E}[|\mathcal{G}|^2] = |\vec{G}|^2$ and $\mathbb{E}[\mathcal{S}] = \text{tr}(\Sigma)$. 

To evaluate the noise scale of value learning and policy gradient we trained our dual network model (described fully in section \ref{sec:dna}) on the three Atari games from the Atari-3 validation set.\footnote{See our parallel Atari-5 paper, included in the supplementary material.} Like \cite{mccandlish2018empirical}, we found it necessary to smooth out noise by maintaining an exponential moving average of $|\gbl|^2$. We used $\bbs=16$ and $\bbb=16,384$ in all our experiments. 


\begin{figure}[t]
    \centering
    \includegraphics[width=0.95\textwidth]{figures/NoiseScale.pdf}
    \caption{Noise scale for the two learning objectives.%
   
    For clarity all values are further smoothed using an exponential rolling average (with non-smoothed values presented in a lighter shade). Smoothing may introduce bias, as while $|\mathcal{G}|^2$ and $\mathcal{S}$ are unbiased estimators, their ratio may not be.}
    \label{fig:noise_scale}
\end{figure}

The results are presented in Figure \ref{fig:noise_scale}. We found that policy noise ($\sigma_\pi$) reduced by about half during training and that value noise ($\sigma_v$) remained constant. Both noise levels varied very little between the three environments. However, a large variation was observed between the value loss and policy gradient, consistent between environments. The noise levels, measured at the end of training and averaged over all three environments, were found to be $\sigma_\pi=220.5$ and $\sigma_V=17.5$, representing a $12.6$ times difference.

\section{Dual Network Architecture}
\label{sec:dna}

Based on our experimental results in Section \ref{sec:noise} we propose an architecture which takes into account the large difference in noise levels between the two tasks. This architecture consists of three improvements to PPO. First, to reduce negative interference from the noisy policy gradient, policy and value should be learned by independent networks with different hyperparameters (e.g. training epochs and mini-batch size). Second, the variance/bias trade-off in the return estimations should be calibrated to the properties of each task. Finally, a constrained distillation phase should account for any constructive interference between the tasks. 

\subsection{Independent Networks}




Important to DNA is the use of a dual network architecture.\footnote{By dual network, we mean two independent networks. Not to be confused with a dual-head network (which we refer to as a joint network), or with dualing networks \cite{wang2016dueling}.} Not only does this setup allow for learning of the value and policy without destructive interference, it also enables a specialized set of hyperparameters to be calibrated for the distinct tasks of policy and value learning. %
Specifically, the calibration of the mini-batch size, which it has been suggested is best set roughly proportional to $\sigma$ \cite{mccandlish2018empirical}. The policy network outputs a policy $\pi$ and a value estimate $V_\pi$, whereas the value network only outputs a value $V_V$, as depicted in Figure \ref{fig:dna_arch}.



\begin{figure}[t]
    \centering
    \includegraphics[width=0.5\textwidth]{figures/dna_arch.pdf}
    \caption{Architecture of DNA (left) compared to a single network setup (right).}
    \label{fig:dna_arch}
\end{figure}


\subsection{Decoupled Return Estimation}

To better facilitate control over the noise levels for policy and value learning, DNA makes use of two different return estimations, both using TD($\lambda$) \cite{sutton1988learning} as follows
\begin{align}
    V_\text{targ}(s_t) &:= \text{TD}^{(\gamma, \lambda_V)}(s_t), \\
    V_\text{adv}(s_t) &:= \text{TD}^{(\gamma, \lambda_\pi)}(s_t),
\end{align}

where $\lambda_V$ and $\lambda_\pi$ are hyperparameters controlling the variance / bias trade-off of each estimate. The $V_\text{targ}$ estimates are used as targets for training the value function, whereas $V_\text{adv}$ are used for advantage estimates used for policy gradient estimates given by


\begin{align}
\hat{A}_t &:= V_\text{adv}(s_t) - V(s_t).
\end{align}

This formulation of the advantages is equivalent to the general advantage estimation (GAE) \cite{schulman2015high}, that is, for any $\lambda \in [0..1], \gamma \in [0..1)$  we have,\footnote{A proof for this claim is provided in Appendix \ref{app:td}}
\begin{align}
    \hat{A}_t^{\text{GAE}(\gamma, \lambda)} &= \text{TD}^{(\gamma, \lambda)}(s_t) - V(s_t).
\end{align}

Because of this, it is common when using PPO to generate value targets, $V_\text{targ}$, for the value function by adding the network's value estimates to the (unnormalized) advantages, implicitly settings $\lv=\lp$.\footnote{For example, see line 65 of the popular baselines implementation of PPO. \url{https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/runner.py} }  We hypothesize that due to bootstrapping, estimates used for \vt benefit from being \textit{low bias}, conversely, because estimates used for advantage estimation contribute to the noise of the policy gradient, they would benefit from being \textit{low variance}. This suggests $\lp < \lv$, a thought which we test empirically in section \ref{sec:results_re}.

\subsection{Distillation}
\label{sec:distillation}

Because value learning is a simpler task in terms of noise level, it makes sense to try to transfer some of the knowledge learned by the value network to the policy network. If the value network learns to identify important features in the environment, we would like the policy network to identify those features also. To facilitate this, we employ a constrained distillation update \cite{hinton2015distilling}. Distillation between two identical networks has the property that the student's target function since an identical network generated it, is guaranteed to be in the model's solution space. Also, because the distillation targets are deterministic,\footnote{That is to say, $V_\text{targ}(s_t)$ are samples drawn from the random variable $R(s_t)$, where $R$ is a (noisy) return estimator, whereas $V(s_t)$ is a deterministic function that produces an estimate of $\mathbb{E}[R(s_t)]$} the process is also very low noise, which we verify in Appendix \ref{app:distil_noise}.

We contrast this with the auxiliary update of PPG which trains both the value function and the policy on an auxiliary task. This auxiliary task equates to learning $\frac{1}{N}\sum_{i=0}^{N}{V_\text{targ}^{\pi_{-i}}}$ where $V_\text{targ}^{\pi_{-i}}$ is a value estimate for the policy from $i$ updates prior. For quickly changing policies, and large $N$, these updates may cause tension with the value estimates of the current policy.\footnote{A better way to do auxiliary tasks would be to train a separate value network output head, dedicated to the auxiliary task. This way the agent could learn the value of the current policy, along with the moving average value. We believe, however, that the better solution to overfitting the value function to recent data is simply to train less. An idea which we discuss further in Section \ref{sec:discussion}.}

We use $V_V(s)$ as the distillation targets, with the input states $s$ being taken from the current rollout. However, unlike policy and value training, distillation state inputs need not be generated on-policy. We explored other distillation targets, covered in Appendix \ref{app:distil_targets}, but found $V_V(s)$ to be the best of those tried. Distillation is performed, like PPG's auxiliary task, using mean squared error, and under a soft constraint on the policy network's policy, specifically

\begin{equation}
    L_t^{D}(\theta) := \mathbb{\hat{E}}_t \big[(V_\pi(s_t)-V_V(s_t))^2 \big] +
    \beta \cdot \mathbb{\hat{E}}_t\large[ \text{KL}(\pi_\text{old}(\cdot|s_t), \pi(\cdot|s_t)) \big]
\end{equation}

where $\beta$ is the policy constraint coefficient, and $\pi_\text{old}$ is a copy of the policy before the distillation update. During updates gradients are only propagated through the policy network, and not the value network.


\subsection{Training}

Like Phasic Policy Gradient (PPG) \cite{cobbe2021phasic}, DNA splits training into three distinct phases, but unlike PPG, rather than using a large replay buffer, we perform all updates on-policy on the current batch of rollout data. Each of the three phases optimizes a single objective, for some number of epochs, using its own optimizer, with a unique set of hyperparameters. The optimization objective for the policy network of DNA is the clipped surrogate object from PPO \cite{schulman2017proximal} including the entropy bonus 
\begin{equation}
    L^{\text{CLIP}}_t := \hat{\mathbb{E}}_t \bigg[ \text{min}( \rho_t(\theta)\hat{A_t}, \text{clip}(\rho_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t) + c_\text{eb} \cdot \text{S}[\pi(s_t)] \bigg]
\end{equation}

where $\text{S}$ is the entropy in nats, $\rho_t$ is the ratio $\frac{\pi(a_t|s_t)}{\pi_\text{old}(a_t|s_t)}$ at time $t$, $\epsilon$ is the clipping coefficient, and $c_\text{eb}$ is the entropy bonus coefficient. For value loss we use the squared-error loss, 
\begin{equation}
    L_t^{VF} := \mathbb{\hat{E}}_t \big[ (V_V(s_t)-V_{\text{targ}}(s_t))^{2} \big].
\end{equation}

In summary, the DNA algorithm separates the tasks of policy learning and value learning into a network designed to handle the high noise of policy learning and a separate network designed to handle the lower noise task of value learning. Knowledge from the value learning network is transferred to the policy network through a separate constrained distillation phase, which allows for constructive interference between the learning tasks while minimizing the destructive. We formalize the algorithm as follows.

\begin{algorithm}[H]
    \label{alg:ppo_dna}
    \caption{Proximal Policy Optimization with Dual Network Architecture}
    \begin{algorithmic}[1]
    \Procedure{PPO-DNA}{}\\
    
   
   
    
    \For{$t = 1$ to $N$}
        \For{$i = 1$ to $A$}
            \State{Run policy $\pi_t$ in environment $i$ for $T$ timesteps}
        \EndFor
        
        \State{Compute $V_\text{targ} \gets \text{TD}^{(\gamma, \lambda_V)}$}
        
        \State{Compute $\hat{A} \gets \text{TD}^{(\gamma, \lambda_\pi)} - V_\text{adv}$}
    
       
        \For{$i = 1$ to $E_\pi$}
            \State{Optimize $L^{CLIP}$ wrt $\theta_{\pi}$}
        \EndFor
    
       
        \For{$i = 1$ to $E_V$}
            \State{Optimize $L^{VF}$ wrt $\theta_V$}
        \EndFor
        
        \State{$\pi_\text{old} \gets \pi_t $ }
    
       
        \For{$i = 1$ to $E_D$}
            \State{Optimize $L^{D}$ wrt $\theta_\pi$}
        \EndFor
    \EndFor
    \EndProcedure
    \end{algorithmic}
\end{algorithm}

\section{Evaluation}
\label{sec:evaluation}



To evaluate the performance of our algorithm, we used the Atari-5 benchmark. Scores on this subset, when properly weighted, correlate well with the median score performance of an algorithm if it had been run on all 57-games. In all cases, we fit hyperparameters to the 3-game validation set and only used the 5-game test set for final evaluations. To verify the statistical significance of our results, we performed multiple seeded runs.

We opted for the more difficult stochastic ALE settings recommended as best practice by \cite{machado2018revisiting}. However, to better understand our results in the context of prior work, we also provide detailed results under the simpler deterministic settings in Appendix \ref{app:easy}, and additionally provide for reference a single seed evaluation on the full 57-game set in Appendix \ref{app:full}. Unless otherwise specified, agents were scored according to their average performance over the previously completed 100-episodes at the end of training.

A coarse hyperparameter sweep found initial hyperparameters for our model on the Atari-3 validation set. Notably, we found the optimal mini-batch size for policy and distillation to be the minimum we tested (256), while the optimal mini-batch size for policy was the largest tested (2048). For optimization, we used Adam \cite{kingma2014adam}, over the standard 200 million frames. Full hyperparameter details for our experiments are given in Appendix \ref{app:hyperparameters}.

In order to understand the impact of the hyperparameters introduced by our algorithm, namely $E_\pi, E_V, E_D$ and $\lambda_\pi, \lambda_V$, we performed the following experiments on the Atari-3 validation set. We started by setting $E_\pi = E_V = E_D = 2$, then searched over $E_V \in [1,2,3,4]$, and selected the best $E_V$. We then searched over $E_\pi$ and $E_D$ while keeping the other two parameters constant. We also performed a similar experiment on the impact of $\lambda_V$ and $\lambda_\pi$ by setting $\lambda_V$ to 0.95 then sweeping across $\lambda_\pi \in [0.6, 0.8, 0.9, 0.95, 0.975]$, then setting $\lambda_V$ to the best $\lambda_\pi$ and repeating. This process allowed us to verify if the best settings occur at $\lambda_V=\lambda_\pi$ or at $\lambda_V \neq \lambda_\pi$. During our $\lambda$ tuning, we also recorded noise levels for one of the seeds, the results of which we discuss in Section \ref{sec:results_re}.

We evaluated our algorithm DNA, against PPO, and PPG. All models used the `NatureCNN' encoder from \cite{mnih2015human}. However, because DNA and PPG use two networks and thus twice the parameters, we doubled the number of channels of the PPO encoder which we refer to as PPO (2x) and which is very similar to the encoder used by \cite{badia2020agent57}. All models have have approximately 3.5 million parameters in total. For fairness we repeated the same sweep on PPO for the number of training epochs ($E_\text{PPO}$) and $\lambda$ used for GAE ($\lambda_\text{PPO}$). To verify that our settings did not inadvertently degrade the performance of PPO we also tested against the settings used by \cite{schulman2017proximal}, and refer to this as PPO (original). These results are included in the ablation study in Appendix \ref{app:ablation}.

\section{Main Experimental Results}
\label{sec:results}

In this section we present the results of our main study comparing PPO to DNA and PPG on Atari-5.

\subsection{Impact of Training Epochs}

We found that using \textit{less} than three policy epochs for DNA dramatically increased the performance of the agent while also decreasing the computation required to train the agent (Figure \ref{fig:dna_tuning}). We also found that DNA was robust to the choice of value and distillation epochs but that overtraining on value ($E_V=4$) marginally decreased performance. This was not the case for distillation updates. The distillation phase demonstrated some benefit as indicated by the $E_D=0$ results under-perform the others. We selected $E_\pi=2, E_V=1, E_D=2$ as the optimal settings and used these for the remainder of the experiments. These settings require a total of four updates for the policy network, but only one update for the value network. The training curves also suggest that DNA may benefit from training beyond the standard 200-million frames. 

\begin{figure}[t]
    \centering
    \includegraphics[width=0.32\textwidth]{figures/tunning_dna_epochs_v.pdf}
    \includegraphics[width=0.32\textwidth]{figures/tunning_dna_epochs_pi.pdf}
    \includegraphics[width=0.32\textwidth]{figures/tunning_dna_epochs_d.pdf}
    \caption{Training curves for DNA with various epochs. The DNA algorithm is most sensitive to the number of policy epochs, and is optimal on our validation set with $E_\pi=2$. Applying policy updates 3 or 4 times significantly reduces the agents performance.}
    \label{fig:dna_tuning}
\end{figure}

\subsection{Return Estimation}
\label{sec:results_re}

We found that the choice of $\lambda_\pi$ made a large difference to the performance of our agent. As was hypothesised the optimal settings for $\lambda_\pi$ differed to those of $\lambda_V$ with $\lambda_\pi$ preferring a lower variance, higher bias value of $0.8$, while the value targets, $\lambda_V$ preferred a lower bias, higher variance setting of $0.95$. Our results are presented in Figure \ref{fig:dna_lambda}. Notable is that the non-homogeneous setting $\lambda_V=0.95, \lambda_\pi=0.8$ outperformed both homogeneous choices $\lambda_\pi=\lambda_V=0.8$ and $\lambda_\pi=\lambda_V=0.95$. We found, as expected, that noise scale can be reduced by selecting a low value for $\lambda_\pi$. However, against our expectations, setting $\lambda_V$ lower actually increased, not decreased the noise level of the value learning task. Once we discovered this we ran addition experiments for a broader choice of $\lambda_V$, and include the results in Figure \ref{fig:dna_lambda}.


\begin{figure}
    \centering
    \includegraphics[height=1.1in]{figures/dna_lambda.pdf}
    \includegraphics[height=1.1in]{figures/noise_policy.pdf}
    \includegraphics[height=1.1in]{figures/noise_value.pdf}
    \caption{Performance of DNA across a range of $\lambda$ values for return estimations. Left: The optimal settings for $\lambda_V$ and $\lambda_\pi$ differed. Shading indicates standard error over 5 seeds. Mid/right: $\sigma_\pi$ increased with $\lambda_\pi$ where as $\sigma_V$ decreased with $\lambda_V$. Shading indicates standard deviation over the environments.}
    
    \label{fig:dna_lambda}
\end{figure}

\subsection{Comparision to PPO and PPG}

We found DNA to outperform both PPO and PPG on the Atari-5 dataset by a wide margin (Figure \ref{fig:final_plot}). We give Atari-5 summary scores, as well as training plots for each individual game. We also include references for Rainbow DQN, although note that these scores were generated under the simpler deterministic settings.

\begin{figure}[t]
    \centering
    $\vcenter{\hbox{
    \includegraphics[height=1.5in]{figures/final_plot.pdf}}}$
    $\vcenter{\hbox{
    \includegraphics[height=1.1in]{figures/final_individual.pdf}
    }}$
    \caption{Left: Results on the Atari-5 benchmark. DNA outperforms PPO by a wide margin, and even exceeds the Rainbow algorithm despite the more difficult settings used in our experiment. Right: Individual training curves for each of the five games.}
    \label{fig:final_plot}
\end{figure}

\section{Discussion}
\label{sec:discussion}

\paragraph{Reduced Training as an Alternative to Experience Replay}

In this subsection, we discuss the surprising result that \textit{less} training, both on policy and value updates, typically improves the agent's performance. One explanation for this could be, as \cite{cobbe2021phasic} have suggested, that in the value case, this might be due to overfitting to recent experience.\footnote{This is plausible, especially with high variance returns} In terms of policy updates, where the largest performance degradation was observed, this could be due to increased training epochs causing large changes in policy, which may suggest a better constraint than PPO's clipping is required.\footnote{It is worth noting that PPO does clip the entropy bonus, and therefore updates with a non-zero entropy bonus could make arbitrarily large changes to the policy regardless of the choice of $\epsilon$.} That being said, limiting the training epochs to one or two provides straightforward solution to the problem while having the additional benefit of decreasing the computational resources required to train an agent. Even in Deep Q-learning, others have found that, beyond some point, increasing the number of times an example is processed has a negative effect unless the learning rate is proportionally reduced  \cite{aitchison2019optimal}.

\paragraph{Choice of $\lambda$}

Our results have shown strong evidence for the benefit of selecting $\lambda_{\pi} < \lambda_V$. As expected, picking \textit{lower} values of $\lambda_\pi$ reduces the policy gradient noise. However, counterintuitively, noise on value learning reduces with \textit{higher} choices of $\lambda_V$. That is, policy learning prefers low \textit{variance}, but value learning prefers low \textit{bias}. This result may explain why clipping, which introduces bias, performs poorly on value learning but quite well on policy learning.\footnote{By value clipping, we mean the value trust-region suggested by \cite{schulman2015high}, but which has been shown hurt performance by \cite{andrychowicz2020matters}.} This result also suggests that an even higher choice of $\lambda_V$ than the $0.95$ we used in our experiments may be appropriate.

\paragraph{Performance of PPG on Atari}

Our algorithms shares a lot in common with the PPG algorithm, which has shown strong results on the Procgen \cite{cobbe2020leveraging} environment. However, PPG performed quite poorly on the Atari games tested, only marginally outperforming the PPO reference. We suspect that this is due to the auxiliary task involving learning stale values estimates. We verify in the Appendix \ref{app:ablation} that the poor performance was not due to hyperparameter differences.

\paragraph{Performance of PPO on Atari}
Perhaps the most surprising result of our study was that PPO could perform well on Atari by simply increasing the mini-batch size, and reducing the training epochs down to 1. These settings resulted in an algorithm that was 4-times faster to train than the settings used by \cite{schulman2017proximal}, and produce much stronger results (Appendix \ref{app:ablation}).

\paragraph{Limitations}

Our paper's primary focus was improving PG results on discrete action vision-based problems. We do not cover continuous control problems and leave this for future work. 

\paragraph{Broader Impact}

In our work, we have shown that deep actor-critic reinforcement learning models can effectively solve complex vision-based, discrete-action environments, even without the use of large replay buffers. Surprisingly this can be achieved using \textit{less} not more policy and value network updates. Because Q-learning approaches generally produce deterministic policies, algorithms based on them may `lock in' to a decision based on minimal differences in outcome. This is not always the case with stochastic policies, which are able to randomly split over actions of roughly equal quality. This may lead to fairer outcomes. We do not foresee any direct negative societal impacts from this research. 

\paragraph{Future work} Our algorithm is already highly competitive with Rainbow DQN, and could likely be extended with orthogonal improvements such as adding recurrence \cite{hausknecht2015deep}, distributional value learning \cite{bellemare2017distributional}, residual encoders \cite{espeholt2018impala}, better exploration bonuses \cite{burda2018exploration}, and making use of a learned model \cite{schrittwieser2020mastering}.

\section{Conclusions}


In this paper, we have highlighted noise level as a key difference between policy and value learning. We have introduced an algorithm that accounts for this order-of-magnitude difference by limiting \textit{negative} interference through two independent networks for value and policy but retaining \textit{constructive} interference through a constrained distillation process. Furthermore, we have shown that the variance/bias tradeoff differs for value learning and policy gradient and that return estimation should cater for this. Together these changes result in a novel algorithm, DNA, that outperforms its predecessor PPO, and even surpasses the popular Q-learning approach Rainbow DQN, while under more challenging environmental settings.


\newpage

