---
abstract: 'Recent work has proposed various adversarial losses for training generative adversarial networks. Yet, it remains unclear what certain types of functions are valid adversarial loss functions, and how these loss functions perform against one another. In this paper, we aim to gain a deeper understanding of adversarial losses by decoupling the effects of their component functions and regularization terms. We first derive some necessary and sufficient conditions of the component functions such that the adversarial loss is a divergence-like measure between the data and the model distributions. In order to systematically compare different adversarial losses, we then propose DANTest—a new, simple framework based on discriminative adversarial networks. With this framework, we evaluate an extensive set of adversarial losses by combining different component functions and regularization approaches. This study leads to some new insights into the adversarial losses. For reproducibility, all source code is available at <https://github.com/salu133445/dan>.'
bibliography:
- 'ref.bib'
---

Introduction {#sec:intro}
============

Generative adversarial networks (GANs) [@goodfellow2014] are a class of unsupervised machine learning algorithms. In essence, a GAN learn a generative model with the guidance of another discriminative model which is trained jointly. However, the idea of adversarial losses is not limited to unsupervised learning. Adversarial losses can also be applied to supervised and semi-supervised scenarios (e.g., [@isola2017; @dossantos2017]). Over the past few years, adversarial losses have advanced the state of the art in many fields [@goodfellow2016].

Despite the success, there are several open questions that need to be addressed. On one hand, although plenty adversarial losses have been proposed, we have little theoretical understanding of what makes a loss function a valid one.

On the other hand, we note that any two adversarial losses can differ in terms of not only the *component functions* (e.g., minimax or hinge; see [Section \[sec:background\]]{}) used in the main loss function that sets up the two-player adversarial game, but also the *regularization approaches* (e.g., gradient penalties [@gulrajani2017]) used to regularize the models. However, it remains unclear how they respectively contribute to the performance of an adversarial loss. In other words, when empirically compare two adversarial losses, we need to decouple the effects of the component functions and the regularization terms, otherwise we cannot tell which one of them makes an adversarial loss better than the other.

Among existing comparative analysis of adversarial losses, to the best of our knowledge, only @lucic2018 and @kurach2018 attempted to decouple the effects of the component functions and regularization approaches. But, only few combinations of component functions and regularization approaches were tested in these two prior works, only seven and nine respectively. We attribute this to the high computational cost that may involve to conduct the experiments, and, more importantly, the lack of a framework to systematically evaluate adversarial losses.

                                             $f$                     $g$                       $h$                 $y^*$
  --------------------------------- --------------------- ------------------------- ------------------------- ---------------
  minimax [@goodfellow2014]          $-\log(1 + e^{-y})$   $-y - \log(1 + e^{-y})$   $-y - \log(1 + e^{-y})$        $0$
  nonsaturating [@goodfellow2014]    $-\log(1 + e^{-y})$   $-y - \log(1 + e^{-y})$     $\log(1 + e^{-y})$           $0$
  Wasserstein [@arjovsky2017wgan]            $y$                    $-y$                      $-y$                  $0$
  least squares [@mao2017]              $-(y - 1)^2$               $-y^2$                  $(y - 1)^2$         $\frac{1}{2}$
  hinge [@lim2017; @tran2017]         $\min(0, y - 1)$        $\min(0, -y - 1)$               $-y$                  $0$

These two research questions can be summarized as follows:

1.  What certain types of component functions are theoretically valid adversarial loss functions?

2.  How different combinations of the component functions and the regularization approaches perform empirically against one another?

We aim to tackle these two RQs in this paper to advance our understanding of the adversarial losses. Specifically, our contribution to RQ1 is based on the intuition that a favorable adversarial loss should be a divergence-like measure between the distribution of the real data and the distribution of the model output, since in this way we can use the adversarial loss as the training criterion to learn the model parameters. We derive necessary and sufficient conditions such that an adversarial loss has such a favorable property (Sections \[sec:necessary\_conditions\] and \[sec:sufficient\_conditions\]). Interestingly, our theoretical analysis leads to a new perspective to understand the underlying game dynamics of adversarial losses ([Section \[sec:psi\_function\_analysis\]]{}).

For RQ2, we need an efficient way to compare different adversarial losses. Hence, we adopt the discriminative adversarial networks (DANs) [@mirza2014], which are essentially conditional GANs with both the generator and the discriminator being discriminative models. Based on DANs, we propose *DANTest*—a new, simple framework for comparing adversarial losses ([Section \[sec:dantest\]]{}). The main idea is to first train a number of DANs for a supervised learning task (e.g., classification) using different adversarial losses, and then compare their performance using standard evaluation metrics for supervised learning (e.g., classification accuracy). With the DANTest, we systematically evaluate 168 adversarial losses featuring the combination of ten existing component functions, two new component functions we originally propose in this paper in light of our theoretical analysis, and 14 existing regularization approaches ([Section \[sec:experiments\]]{}). Moreover, we use the DANTest to empirically study the effect of the Lipschitz constant [@arjovsky2017wgan], penalty weights [@mescheder2018], momentum terms [@kingma2014], and others. We discuss the new insights that are gained, and their implications to the design of adversarial losses in future research.

Background {#sec:background}
==========

Generative Adversarial Networks {#sec:gan}
-------------------------------

A generative adversarial network [@goodfellow2014] is a generative latent variable model that aims to learn a mapping from a latent space $\mathcal{Z}$ to the data space $\mathcal{X}$, i.e., a generative model $G$, which we will refer to as the *generator*. A discriminative model $D$ (i.e., the *discriminator*) defined on $\mathcal{X}$ is trained alongside the $G$ to provide guidance for it. Let $p_d$ denote the *data distribution* and $p_g$ be the *model distribution* implicitly defined by $G({\mathbf{z}})$ when ${\mathbf{z}}\sim p_{\mathbf{z}}$. In general, most GAN loss functions proposed in the literature can be formulated as: $$\begin{aligned}
  \label{eq:discriminator}
  \max_{D}\;&{\mathbb{E}}_{{\mathbf{x}}\sim p_d}[f(D({\mathbf{x}}))] + {\mathbb{E}}_{{\tilde{{\mathbf{x}}}}\sim p_g}[g(D({\tilde{{\mathbf{x}}}}))]\,,\\
  \label{eq:generator}
  \min_{G}\;&{\mathbb{E}}_{{\tilde{{\mathbf{x}}}}\sim p_g}[h(D({\tilde{{\mathbf{x}}}}))]\,,\end{aligned}$$ where $f$, $g$ and $h$ are real functions defined on the data space (i.e., ${\mathcal{X}}\to {\mathbb{R}}$) and we will refer to them as the *compoenent functions*. We summarize in [ \[tab:loss\_functions\]]{} the component functions $f$, $g$ and $h$ used in some existing adversarial losses.

                                                  $p_{{\hat{{\mathbf{x}}}}}$              $R(x)$
  --------------------------------------------- ------------------------------ -----------------------------
  coupled gradient penalties [@gulrajani2017]    $p_d + U[0, 1]\,(p_g - p_d)$   $(x - k)^2$ or $\max(x, k)$
  local gradient penalties [@kodali2017]              $p_d + c\,N(0, I)$        $(x - k)^2$ or $\max(x, k)$
  R~1~ gradient penalties [@mescheder2018]                  $p_d$                           $x$
  R~2~ gradient penalties [@mescheder2018]                  $p_g$                           $x$

  -------------------------------------------------- ------------------------------------------------ --------------------------------------------- ---------------------------------------------
   ![image](coupled_gp.png){width=".135\linewidth"}   ![image](local_gp.png){width=".135\linewidth"}   ![image](r1_gp.png){width=".135\linewidth"}   ![image](r2_gp.png){width=".135\linewidth"}
           \(a) coupled gradient penalties                    \(b) local gradient penalties                   \(c) R~1~ gradient penalties                  \(d) R~2~ gradient penalties
  -------------------------------------------------- ------------------------------------------------ --------------------------------------------- ---------------------------------------------

Some prior work has also investigated the so-called IPM-based GANs, where the discriminator is trained to estimate an integral probability metric (IPM) between $p_d$ and $p_g$: $$\begin{aligned}
  \label{eq:ipm_distance}
  d(p_d, p_g) = -\sup_{D\in\mathcal{D}}\;{\mathbb{E}}_{{\mathbf{x}}\sim p_d}[D({\mathbf{x}})] + {\mathbb{E}}_{{\tilde{{\mathbf{x}}}}\sim p_g}[D({\tilde{{\mathbf{x}}}})]\,,\end{aligned}$$ where $\mathcal{D}$ is a set of functions from ${\mathcal{X}}$ to ${\mathbb{R}}$. For example, the Wasserstein GANs [@arjovsky2017wgan] consider $\mathcal{D}$ to be the set of all 1-Lipschitz functions. Other examples include McGAN [@mroueh2017mcgan], MMD GAN [@li2017] and Fisher GAN [@mroueh2017fishergan]. Please note that the main difference between and is that in the latter we constrain $D$ to be in some set of functions $\mathcal{D}$.

Gradient Penalties {#sec:gradient_penalties}
------------------

As the discriminator is often found to be too strong to provide reliable gradients to the generator, one regularization approach is to use some gradient penalties to constrain the modeling capability of the discriminator. Most gradient penalties proposed in the literature take the following form: $$\label{eq:gradient_penalties}
  \lambda\,{\mathbb{E}}_{{\hat{{\mathbf{x}}}}\sim p_{{\hat{{\mathbf{x}}}}}}[R(||\nabla_{{\hat{{\mathbf{x}}}}} D({\hat{{\mathbf{x}}}})||)]\,,$$ where the *penality weight* $\lambda \in {\mathbb{R}}$ is a pre-defined constant, and $R(\cdot)$ is a real function. The distribution $p_{{\hat{{\mathbf{x}}}}}$ defines where the gradient penalties are enforced. [ \[tab:gradient\_penalties\]]{} shows the distribution $p_{{\hat{{\mathbf{x}}}}}$ and function $R$ used in some common gradient penalties. And, [ \[fig:gradient\_penalties\]]{} illustrates $p_{{\hat{{\mathbf{x}}}}}$.

When gradient penalties are enforced, the loss function for training the discriminator contains not only the component functions $f$ and $g$ in but also the *regularization term* .

Spectral Normalization {#sec:spectral_normalization}
----------------------

Another regularization approach we consider is the spectral normalization proposed by @miyato2018. It normalizes the spectral norm of each layer in a neural network to enforce the Lipschitz constraints. While the gradient penalties introduced in [Section \[sec:gradient\_penalties\]]{} impose local regularizations, the spectral normalization imposes a global regularization on the discriminator. Therefore, it is possible to combine the spectral normalization with the gradient penalties. We will examine this in [Section \[sec:exp\_adversarial\_losses\]]{}.

Theoretical Results {#sec:theory}
===================

In the following analysis, we follow the notations in and . Proofs can be found in [Appendix \[app:sec:proofs\]]{}.

Favorable properties for adversarial losses
-------------------------------------------

Let us first consider the minimax formulation: $$\begin{aligned}
  \label{eq:minimax}
  \min_{G}\;\max_{D}\;&{\mathbb{E}}_{{\mathbf{x}}\sim p_d}[f(D({\mathbf{x}}))] + {\mathbb{E}}_{{\tilde{{\mathbf{x}}}}\sim p_g}[g(D({\tilde{{\mathbf{x}}}}))]\,.\end{aligned}$$ We can see that if the discriminator is able to reach optimality, the training criterion for the generator is $$\begin{aligned}
  \label{eq:g_loss}
  L_G &= \max_{D}\;{\mathbb{E}}_{{\mathbf{x}}\sim p_d}[f(D({\mathbf{x}}))] + {\mathbb{E}}_{{\tilde{{\mathbf{x}}}}\sim p_g}[g(D({\tilde{{\mathbf{x}}}}))]\,.\end{aligned}$$

In general, for a valid adversarial loss, the discriminator is responsible for providing a measure of the discrepancy between the data distribution $p_d$ and the model distribution $p_g$. In principle, this will then serve as the training criterion for the generator to push $p_g$ towards $p_d$. Hence, we would like such an adversarial loss to be a divergence-like measure between $p_g$ and $p_d$. From this view, we can now define the following two favorable properties of adversarial losses.

[(Weak favorable property)]{.nodecor} For any fixed $p_d$, $L_G$ has a global minimum at $p_g = p_d$. \[prop:weak\]

[(Strong favorable property)]{.nodecor} For any fixed $p_d$, $L_G$ has a unique global minimum at $p_g = p_d$. \[prop:strong\]

We can see that [Property \[prop:strong\]]{} makes $L_G - L^*_G$ a divergence of $p_d$ and $p_g$ for any fixed $p_d$, where $L^*_G = L_G\,\big\rvert_{\,p_g = p_d}$, and [Property \[prop:weak\]]{} provides a weaker version when the identity of indiscernibles is not necessary. Note that $L_G$ is not a divergence since $L_G \geq 0$ does not always hold.

$\Psi$ and $\psi$ functions
---------------------------

In order to derive some necessary and sufficient conditions for Properties \[prop:weak\] and \[prop:strong\], we first observe from that $$\begin{aligned}
  &L_G = \max_{D}\;\int_{\mathbf{x}}p_d({\mathbf{x}})\,f(D({\mathbf{x}})) + p_g({\mathbf{x}})\,g(D({\mathbf{x}}))\,d{\mathbf{x}}\\
  \begin{split}
    &= \int_{\mathbf{x}}(p_d({\mathbf{x}}) + p_g({\mathbf{x}}))\\
    &\quad\;\;\, \max_{D} \left(\frac{p_d({\mathbf{x}})\,f(D({\mathbf{x}}))}{p_d({\mathbf{x}}) + p_g({\mathbf{x}})} + \frac{p_g({\mathbf{x}})\,g(D({\mathbf{x}}))}{p_d({\mathbf{x}}) + p_g({\mathbf{x}})}\right)\,d{\mathbf{x}}\,.
  \end{split}\end{aligned}$$ Now, if we let $\tilde{\gamma} = \frac{p_d({\mathbf{x}})}{p_d({\mathbf{x}}) + p_g({\mathbf{x}})}$ and $\tilde{y} = D({\mathbf{x}})$, we get $$\label{eq:g_loss_expanded}
  \begin{split}
    L_G &= \int_{\mathbf{x}}(p_d({\mathbf{x}}) + p_g({\mathbf{x}}))\\
    &\quad\qquad\; \max_{\tilde{y}}\;\tilde{\gamma}\,f(\tilde{y}) + (1 - \tilde{\gamma})\,g(\tilde{y})\,d{\mathbf{x}}\,.
  \end{split}$$ Please note that $\tilde{\gamma}({\mathbf{x}}) = \frac{1}{2}$ if and only if $p_d({\mathbf{x}}) = p_g({\mathbf{x}})$. Let us now consider the terms inside the integral and define the following two functions: $$\begin{aligned}
  \label{eq:big_psi_function}
  &\Psi(\gamma, y) = \gamma\,f(y) + (1 - \gamma)\,g(y)\,,\\
  \label{eq:small_psi_function}
  &\psi(\gamma) = \max_y\;\Psi(\gamma, y)\,,\end{aligned}$$ where $\gamma \in [0, 1]$ and $y \in {\mathbb{R}}$ are two variables independent of ${\mathbf{x}}$. We visualize in Figures \[fig:psi\_functions\](a)–(d) the $\Psi$ and $\psi$ functions for different common adversarial losses (see [Appendix \[app:sec:psi\_function\_graphs\]]{} for the graphs of the $\psi$ functions alone). These two functions actually reflect some important characteristics of the adversarial losses (see [Section \[sec:psi\_function\_analysis\]]{}) and will be used intensively in our theoretical analysis.

Necessary conditions for the favorable properties {#sec:necessary_conditions}
-------------------------------------------------

For the necessary conditions of Properties \[prop:weak\] and \[prop:strong\], we have the following two theorems.

If [Property \[prop:weak\]]{} holds, then for any $\gamma \in [0, 1]$, $\psi(\gamma) + \psi(1 - \gamma) \geq 2\,\psi(\frac{1}{2})$. \[theo:weak\_necessary\_condition\]

If [Property \[prop:strong\]]{} holds, then for any $\gamma \in [0, 1] \setminus \{\frac{1}{2}\}$, $\psi(\gamma) + \psi(1 - \gamma) > 2\,\psi(\frac{1}{2})$. \[theo:strong\_necessary\_condition\]

With Theorems \[theo:weak\_necessary\_condition\] and \[theo:strong\_necessary\_condition\], we can easily check if a pair of component functions $f$ and $g$ form a valid adversarial loss.

Sufficient conditions for the favorable properties {#sec:sufficient_conditions}
--------------------------------------------------

For sufficient conditions, we have two theorems as follows.

If $\psi(\gamma)$ has a global minimum at $\gamma = \frac{1}{2}$, then [Property \[prop:weak\]]{} holds. \[theo:weak\_sufficient\_condition\]

If $\psi(\gamma)$ has a unique global minimum at $\gamma = \frac{1}{2}$, then [Property \[prop:strong\]]{} holds. \[theo:strong\_sufficient\_condition\]

We also have the following theorem for a more specific guideline for choosing the component functions $f$ and $g$.

If $f'' + g'' \leq 0$ and there exists some $y^*$ such that $f(y^*) = g(y^*)$ and $f'(y^*) = -g'(y^*) \neq 0$, then $\psi(\gamma)$ has a unique global minimum at $\gamma = \frac{1}{2}$. \[theo:strong\_sufficient\_condition2\]

By Theorems \[theo:strong\_sufficient\_condition\] and \[theo:strong\_sufficient\_condition2\], we now see that any component function pair $f$ and $g$ that satisfies the prerequisites in [Theorem \[theo:strong\_sufficient\_condition2\]]{} makes $L_G - L^*_G$ a divergence between $p_d$ and $p_g$ for any fixed $p_d$. Interestingly, while such a theoretical analysis has not been done before, it happens that all the adversarial loss functions listed in [ \[tab:loss\_functions\]]{} have such favorable properties. We intend to examine in [Section \[sec:exp\_properties\]]{} empirically the cases when the prerequisites of [Theorem \[theo:strong\_sufficient\_condition2\]]{} do not hold.

In practice, the discriminator often cannot reach optimality at each iteration. Therefore, as discussed by @nowozin2016 [@fedus2018], the objective of the generator is similar to variational divergence minimization (i.e., to minimize a lower bound of some divergence between $p_d$ and $p_g$), where the divergence is estimated by the discriminator.

Loss functions for the generator {#sec:g_loss}
--------------------------------

Intuitively, the generator should minimize the divergence-like measure estimated by the discriminator. We have accordingly $h = g$. However, some prior works have investigated setting $h$ different from $g$. In general, most of these alternative generator losses do not change the solutions of the game and are proposed base on some heuristics. While our theoretical analysis concerns with only $f$ and $g$, we intend to empirically examine the effects of the generator loss function $h$ in [Section \[sec:exp\_g\_loss\]]{}.

Analyzing the adversarial game by the $\Psi$ functions {#sec:psi_function_analysis}
------------------------------------------------------

[ \[fig:psi\_functions\]]{} gives us some new insights regarding the adversarial behaviors of the discriminator and the generator. On one hand, if we follow and consider $\tilde{y} = D({\mathbf{x}})$ and $\tilde{\gamma}({\mathbf{x}}) = \frac{p_d({\mathbf{x}})}{p_d({\mathbf{x}}) + p_g({\mathbf{x}})}$, then the discriminator can be viewed as maximizing $\Psi$ along the $\tilde{y}$-axis. On the other hand, since the generator is trained to push $p_g$ towards $p_d$, it can be viewed as minimizing $\Psi$ along the $\tilde{\gamma}$-axis. In this way, we can see why all these $\Psi$ functions are saddle-shaped and have saddle points at $\gamma = \frac{1}{2}$ (i.e., when $p_d({\mathbf{x}}) = p_g({\mathbf{x}})$).

Ideally, if the discriminator can be trained till optimality, then we will be on the green line, the domain of the $\psi$ function. In this case, the generator can be viewed as minimizing $\Psi$ along the green line (i.e., minizing $\psi$). Note that as $L_G$ is an integral over all possible ${\mathbf{x}}$, such adversarial game is actually being played in a (usually) high dimensional space.

By designing the landscape of $\Psi$, we propose and consider two new losses in our empirical study in [Section \[sec:exp\_adversarial\_losses\]]{}:

-   The *absolute* loss, with $f(y) = -h(y) = -|1 - y|$, $g(y) = -|y|$. Its $\Psi$-landscape is similar to those of the least squares and the hinge losses (see [ \[fig:psi\_functions\]]{}(e)).

-   The *asymmetric* loss, with $f(y) = -|y|$, $g(y) = h(y) = -y$. Its $\Psi$-landscape is similar to that of the Wasserstein loss, but the positive part of $y$ is blocked (see [ \[fig:psi\_functions\]]{}(f)).

DANTest {#sec:dantest}
=======

Discriminative adversarial networks (DANs) [@dossantos2017] are essentially conditional GANs [@mirza2014] where both the generator and the discriminator are discriminative models, as shown in [ \[fig:dan\]]{}. Based on DANs, we propose a new, simple framework, dubbed *DANTest*, for systematically comparing different adversarial losses. Specifically, the DANTest goes as follows:

1.  Build several DANs. For each of them, the generator $G$ takes as input a real sample and outputs a fake label. The discriminator takes as input a real sample with either its true label, or a fake label made by $G$, and outputs a scalar indicating if the “sample–label” pair is real.

2.  Train the DANs with different component loss functions, regularization approaches or hyperparameters.

3.  Predict the labels of test data by the trained models.

4.  Compare the performance of different models with standard evaluation metrics used in supervised learning.

Note that the generator is no longer a generative model in this framework, while the discriminator is still trained by the same loss function to measure the discrepancy between $p_d$ and $p_g$. This way, we can still gain insight into the performance and stability for different adversarial losses. Moreover, although we take a classification task as an example here, the proposed framework is generic and can be applied to other supervised learning tasks as well, as long as the evaluation metrics for that task are well defined.

An extension of the proposed framework is the *imbalanced dataset test*, where we examine the ability of different adversarial losses on datasts that feature class imbalance. This can serve as a measure of the *mode collapse* phenomenon [@che2017mdgan], which is a commonly-encountered failure case in GAN training. By testing on datasets with different levels of imbalance, we can examine how different adversarial losses suffer from the mode collapse problem.

Experiments and Results {#sec:experiments}
=======================

Datasets and Implementation Details {#sec:dataset}
-----------------------------------

All the experiments reported here are done based on the DANTest. If not otherwise specified, we use the MNIST handwritten digits database [@lecun1998], which we refer to as the **standard** dataset. As it is class-balanced, we create two imbalanced versions of it. The first one, referred to as the **imbalanced** dataset, is created by augmenting the training samples for digit ‘0’ by shifting them each by one pixel to the top, bottom, left and right, so that it contains *five* times more training samples of ‘0’ than the standard dataset. Moreover, we create the **very imbalanced** dataset, where we have *seven* times more training samples for digit ‘0’ than the standard dataset. For other digits, we randomly sample from the standard dataset and intentionally make the sizes of the resulting datasets identical to that of the standard dataset. We use the same test set for all the experiments.

We implement $G$ and $D$ as convolutional neural networks (see [Appendix \[app:sec:net\_architectures\]]{} for the network architectures). We use the batch normalization [@ioffe2015] in $G$. If the spectral normalization is used, we only apply it to $D$, otherwise we use the layer normalization [@ba2017] in $D$. We concatenate the label vector to each layer of $D$. For the gradient penalties, we use Euclidean norms and set $\lambda$ to $10.0$ (see ), $k$ to $1.0$ and $c$ to $0.01$ (see [ \[tab:gradient\_penalties\]]{}). We use the Adam optimizers [@kingma2014] with $\alpha = 0.001$, $\beta_1 = 0.0$ and $\beta_2 = 0.9$. We alternatively update $G$ and $D$ once in each iteration and train the model for 100,000 generator steps. The batch size is $64$. We implement the model in Python and TensorFlow [@abadi2016]. We run each experiment for ten runs and report the mean and the standard deviation of the error rates.

                        nonsaturating         Wasserstein              hinge
  ------------------ ------------------- --------------------- ---------------------
  $\epsilon = 0.5$      8.47$\pm$0.36       73.16$\pm$6.36        15.20$\pm$2.46
  $\epsilon = 0.9$      8.96$\pm$0.63       57.66$\pm$5.13          8.94$\pm$0.87
  $\epsilon = 1.0$    **8.25$\pm$0.35**     **5.89$\pm$0.26**     **6.59$\pm$0.31**
  $\epsilon = 1.1$      8.62$\pm$0.45       60.30$\pm$7.61          8.02$\pm$0.35
  $\epsilon = 2.0$      9.18$\pm$0.94       69.54$\pm$5.37        11.87$\pm$0.85

  : Error rates (%) for the $\epsilon$-weighted versions of the nonsaturating, the Wasserstein and the hinge losses (see ) on the standard dataset. Here, $\epsilon = 1.0$ corresponds to the original losses.[]{data-label="tab:exp_properties"}

                                                          **unregularized**       **TCGP**            **TLGP**           **R~1~ GP**         **R~2~ GP**           **SN**           **SN + TCGP**       **SN + TLGP**     **SN + R~1~ GP**    **SN + R~2~ GP**
  ------------------------------------------------------ ------------------- ------------------- ------------------- ------------------- ------------------- ------------------- ------------------- ------------------- ------------------- -------------------
  **classic (M)** ([-@goodfellow2014])                      9.11$\pm$0.63       5.65$\pm$0.27       5.42$\pm$0.17      19.01$\pm$3.73      12.91$\pm$1.13       7.37$\pm$0.52       5.55$\pm$0.37       5.57$\pm$0.28      11.16$\pm$2.66      14.00$\pm$2.49
  **classic (N)** ([-@goodfellow2014])                     26.83$\pm$7.17       5.64$\pm$0.23       5.56$\pm$0.31      14.67$\pm$4.86      13.80$\pm$3.20       8.25$\pm$0.35       5.52$\pm$0.16       5.61$\pm$0.50      12.98$\pm$2.71      13.50$\pm$3.78
  **classic (L)**                                          17.38$\pm$5.16       5.66$\pm$0.36       5.55$\pm$0.16      18.49$\pm$5.51      14.92$\pm$5.20       7.98$\pm$0.36       5.70$\pm$0.36       5.48$\pm$0.29      15.45$\pm$6.54      17.61$\pm$7.60
  **hinge (M)**                                                               **4.83$\pm$0.34**   **4.88$\pm$0.25**                         9.49$\pm$5.30     **6.22$\pm$0.23**                       **5.06$\pm$0.33**    10.62$\pm$2.10      12.91$\pm$4.29
  **hinge (N)**                                            37.55$\pm$20.22    **5.00$\pm$0.24**     4.97$\pm$0.24     **7.34$\pm$1.83**     7.54$\pm$1.31       6.90$\pm$0.33       5.05$\pm$0.22     **5.06$\pm$0.39**    11.91$\pm$4.02      12.10$\pm$4.74
  **hinge (L)** ([-@lim2017; -@tran2017])                  11.50$\pm$5.32       5.01$\pm$0.26     **4.89$\pm$0.18**     8.96$\pm$3.55       7.71$\pm$1.82       6.59$\pm$0.31     **4.97$\pm$0.19**     5.18$\pm$0.27      13.63$\pm$4.13      11.35$\pm$3.40
  **Wasserstein** ([-@arjovsky2017wgan])                    7.69$\pm$0.33       5.04$\pm$0.19       4.92$\pm$0.23      13.89$\pm$20.64    **7.25$\pm$1.19**                         5.50$\pm$0.18       5.76$\pm$0.70      13.74$\pm$5.47      13.82$\pm$4.93
  **least squares** ([-@mao2017])                         **7.15$\pm$0.47**     7.27$\pm$0.44       6.70$\pm$0.44      30.12$\pm$28.43     32.44$\pm$21.05      7.88$\pm$0.45       6.69$\pm$0.25       7.11$\pm$0.37     **9.91$\pm$1.55**    11.56$\pm$4.09
  **relativistic** ([-@jolicoeur-martineau2018])           90.20$\pm$0.00       5.25$\pm$0.25       5.01$\pm$0.31     **8.00$\pm$1.63**     8.75$\pm$5.83       7.14$\pm$0.39       5.35$\pm$0.29       5.25$\pm$0.26     **9.31$\pm$2.01**  
  **relativistic hinge** ([-@jolicoeur-martineau2018])     52.01$\pm$9.38      8.28$\pm$10.26                           8.39$\pm$1.92       7.67$\pm$1.82       6.44$\pm$0.16     **5.02$\pm$0.31**                        12.56$\pm$4.42      12.40$\pm$4.55
  **absolute**                                            **6.69$\pm$0.24**     5.23$\pm$0.29       5.20$\pm$0.26       8.01$\pm$1.96                           6.79$\pm$0.45       5.23$\pm$0.13       5.18$\pm$0.35      10.42$\pm$3.07     **9.93$\pm$2.28**
  **asymmetric**                                            7.81$\pm$0.27                           4.94$\pm$0.14       8.79$\pm$3.18     **7.33$\pm$1.01**   **5.98$\pm$0.40**     5.60$\pm$0.29       5.82$\pm$0.44                         **8.80$\pm$1.18**

Examining the necessary conditions for favorable adversarial loss functions {#sec:exp_properties}
---------------------------------------------------------------------------

As discussed in [Section \[sec:sufficient\_conditions\]]{}, we examine here the cases when the prerequisites in [Theorem \[theo:strong\_sufficient\_condition2\]]{} do not hold. We consider the classic nonsaturating, the Wasserstein and the hinge losses and change the training objective for the discriminator into $$\label{eq:imbalanced_loss}
  \max_{D}\;\epsilon\,{\mathbb{E}}_{{\mathbf{x}}\sim p_d}[f(D({\mathbf{x}}))] + {\mathbb{E}}_{{\tilde{{\mathbf{x}}}}\sim p_g}[g(D({\tilde{{\mathbf{x}}}}))]\,,$$ where $\epsilon \in {\mathbb{R}}$ is a constant. The prerequisites in [Theorem \[theo:strong\_sufficient\_condition2\]]{} do not hold when $\epsilon \neq 1$. We illustrate the $\Psi$ functions of these *$\epsilon$-weighted losses* in [Appendix \[app:sec:psi\_function\_graphs\]]{}.

[ \[tab:exp\_properties\]]{} shows the results for $\epsilon = 0.5$, $0.9$, $1.0$, $1.1$, $2.0$, using the spectral normalization for regularization. We can see that all the original losses (i.e., $\epsilon = 1$) result in the lowest error rates. In general, the error rates increase as $\epsilon$ goes away from $1.0$. Notably, the Wasserstein loss turn out failing with error rates over 50% when $\epsilon \neq 1$.

On different discriminator loss functions {#sec:exp_adversarial_losses}
-----------------------------------------

In this experiment, we aim to compare different discriminator loss functions. Specifically, we evaluate an comprehensive set (in total 168) of different combinations of component functions and regularization approaches.

For the component functions, we consider the classic minimax and the classic nonsaturating losses [@goodfellow2014], the Wasserstein loss [@arjovsky2017wgan], the least squares loss [@mao2017], the hinge loss [@lim2017; @tran2017], the relativistic average and the relativistic average hinge losses [@jolicoeur-martineau2018], as well as the absolute and the asymmetric losses we propose and describe in [Section \[sec:psi\_function\_analysis\]]{}.

For the regularization approaches, we consider the coupled, the local, the R~1~ and the R~2~ gradient penalties (**GP**) and the spectral normalization (**SN**). For the coupled and the local gradient penalties, we examine both the two-side and the one-side versions (see [ \[tab:gradient\_penalties\]]{}). We will use in the captions **OCGP** and **TCGP** as the shorthands for the one-side and the two-side coupled gradient penalties, respectively, and **OLCP** and **TLCP** for the one-side and the two-side local gradient penalties, respectively. We also consider the combinations of the SN with different gradient penalties.

We report in [ \[tab:exp\_adversarial\_losses\]]{} the results for all the combinations and present in [ \[fig:training\_progress\]]{} the training progress for the nonsaturating and the hinge losses. We can see that *there is no single winning component functions and regularization approach across all different settings*. Some observations are:

With respect to the **component functions**—

-   The classic minimax and nonsaturating losses never get the lowest three error rates for all different settings.

-   The hinge, the asymmetric and the two relativistic losses are robust to different regularization approaches and tend to achieve lower error rates.

-   The relativistic average loss outperforms both the classic minimax and nonsaturating losses across all regularization approaches. But, the relativistic average hinge loss does not always outperform the standard hinge loss.

With respect to the **regularization approaches**—

-   The coupled and the local GPs outperform the R~1~ and the R~2~ GPs across nearly all different component functions, no matter whether the SN is used or not.

-   The coupled and the local GPs stabilize the training (see [ \[fig:training\_progress\]]{}) and tend to have lower error rates.

-   The R~2~ gradient penalties achieve lower error rates than the R~1~ gradient penalties. In some cases, they can be too strong and even stop the training early (see [ \[fig:training\_progress\]]{} (a)).[^1]

-   Combining either the coupled or the local GP with the SN usually leads to higher error rates than using the coupled or the local GP only.

-   Similarly, combining either the R~1~ or the R~2~ GP with the SN degrades the result. Moreover, it leads to unstable training (see Figures \[fig:training\_progress\](b) and (d)). This result implies that R~1~ and R~2~ GPs do not work well with the SN.

-   Using the one-side GPs instead of their two-side counterparts increase the error rates by 0.1–9.5%. (We report the results for the one-side GPs in [Appendix \[app:sec:results\]]{} due to page limit.)

We also note that some combinations result in remarkably high error rates, e.g., “least squares loss + R~1~ GP”, “least squares loss + R~2~ GP” and “classic minimax loss + R~1~ GP”.

In sum, according to the overall performance and the robustness to different settings, for the component functions, *we recommend the hinge, the asymmetric and the two relativistic losses*. We note that these functions also feature lower computation costs as all their components functions are piecewise linear (see [ \[tab:loss\_functions\]]{} and [Section \[sec:psi\_function\_analysis\]]{}). For the regularization approaches, *we recommend the two-side coupled and the two-side local gradient penalties*.

We also conduct the imbalanced dataset test (see [Section \[sec:dantest\]]{}) on the two imbalanced datasets described in [Section \[sec:dataset\]]{} to compare the regularization approaches. We use the classic nonsaturating loss. As shown in [ \[tab:exp\_imbalanced\_dataset\]]{}, the error rates increase as the level of imbalance increases. The two-side local GP achieve the lowest error rates across all three datasets. The error rates for the R~1~ and the R~2~ GPs increase significantly when the dataset goes imbalanced.

### Effects of the Lipschitz constants {#sec:exp_lipschitz}

In this experiment, we examine the effects of the Lipschitz constant ($k$) used in the coupled and the local GPs (see [ \[tab:gradient\_penalties\]]{}). We use the classic nonsaturating loss here. We report in [ \[fig:exp\_lipschitz\]]{} the results for $k = 0.01$, $0.1$, $1$, $10$, $100$. We can see that the error rate increases as $k$ goes away from $1.0$, suggesting that $k = 1$ is indeed a good default value. Moreover, the two-side GPs are more sensitive to $k$ than their one-side counterparts.

We note that @petzka2018 suggested that the one-side coupled GP are preferable to the two-side version and showed empirically that the former has more stable behaviors. However, we observe in our experiments that the two-side penalties usually lead to faster convergence to lower error rates compared to the one-side penalties.[^2]

                  standard             imbalanced          very imbalanced
  --------- --------------------- --------------------- ---------------------
  TCGP           5.64$\pm$0.23         7.09$\pm$0.64         8.12$\pm$0.31
  OCGP           7.20$\pm$0.39         8.86$\pm$0.65       10.23$\pm$0.75
  TLGP         **5.51$\pm$0.27**     **6.94$\pm$0.28**     **8.10$\pm$0.55**
  OLGP           6.92$\pm$0.21         8.63$\pm$0.75       10.21$\pm$0.52
  R~1~ GP      14.67$\pm$4.86        18.66$\pm$5.60        27.90$\pm$9.59
  R~2~ GP      13.80$\pm$3.20        15.70$\pm$2.07        29.97$\pm$12.4

  : Error rates (%) for different gradient penalties (using the nonsaturating loss) on datasets with different levels of imbalance.[]{data-label="tab:exp_imbalanced_dataset"}

### Effects of the penalty weights {#sec:exp_penalty_weights}

We then examine the effects of the penalty weights ($\lambda$) for the R~1~ and the R~2~ GPs (see ). We consider the classic nonsaturating, the Wasserstein and the hinge losses. We present in [ \[fig:exp\_r1r2\_penalty\_weight\]]{} the results for $\lambda = 0.01$, $0.1$, $1$, $10$, $100$. We can see that the R~1~ GP tends to outperform the R~2~ GP, while they are both sensitive to the value of $\lambda$. Hence, future research should run hyperparmeter search for $\lambda$ to find out its optimal value. When the spectral normalization is not used, the hinge loss is less sensitive to $\lambda$ than the other two losses. However, when spectral normalization is used, the error rate increases as $\lambda$ increases, which again implies that the R~1~/R~2~ GPs and the SN do not work well together.

\
(a) without the spectral normalization\
\
(b) with the spectral normalization

On different generator loss functions {#sec:exp_g_loss}
-------------------------------------

As discussed in [Section \[sec:g\_loss\]]{}, we also aim to examine the effects of the generator loss function $h(\cdot)$. We consider the classic and the hinge losses for the discriminator and the following three generator loss functions: minimax (**M**)—$h(x) = g(x)$, nonsaturating (**N**)—$h(x) = \log(1 + e^{-x})$, and linear (**L**)—$h(x) = -x$. We report the results in the first six rows of [ \[tab:exp\_adversarial\_losses\]]{}. For the classic discriminator loss, we see no single winner among the three generator loss functions across all the regularization approaches, which implies that the heuristics behind these alternative losses might not be true. For the hinge discriminator loss, the minimax generator loss is robust to different regularization approaches and achieves three lowest and four lowest-three scores. Hence, *we recommend to use hinge loss for the discriminator and minimax loss for the generator* as the overall best choice according to our experimental results.

Effects of the momentum terms of the optimizers {#sec:exp_momentum}
-----------------------------------------------

We observe a trend towards using smaller momentum [@radford2016] or even no momentum [@arjovsky2017wgan; @gulrajani2017; @miyato2018; @brock2018] in GAN training. Hence, we would also like to examine the effects of momentum terms in the optimizers with the proposed framework. As suggested by @gidel2018, we also include a negative momentum value of $-0.5$. We use the classic nonsaturating loss and the SN along with the coupled GPs for regularization. [ \[fig:exp\_momentum\]]{} shows the results for all combinations of $\beta_1 = -0.5$, $0.0$, $0.5$, $0.9$ for $G$ and $D$. We can see that for the two-side coupled GP, using larger momenta in both $G$ and $D$ leads to lower error rates, while there is no specific trend for the one-side coupled GP.

Discussions and Conclusions {#sec:discussions_and_conclusions}
===========================

In this paper, we have shown in theory what certain types of component functions form a valid adversarial loss. We have also introduced a new framework called DANTest for comparing adversarial losses. With DANTest, we systematically compared combinations of different component functions and regularization approaches to decouple their effects. Our empirical results show that there is no single winning component functions or regularization approach across all different settings. Our theoretical and empirical results can together serve as a reference for choosing or designing adversarial training objectives in future research.

As compared to the commonly used metrics for evaluating generative models, such as the Inception Score [@salimans2016] and Fréchet Inception Distance [@heusel2017] adopted in @lucic2018 and @kurach2018, the DANTest is simpler and is easier to control and extend. This allows us to easily evaluate new adversarial losses. However, we note that while the discriminator in a DAN is trained to optimize the same objectives as in a conditional GAN, the generators in the two models actually work in opposite ways ($\mathcal{X} \to \mathcal{Z}$ in a DAN versus $\mathcal{Z} \to \mathcal{X}$ in a GAN). Hence, it is unclear whether the empirical results can be generalized to conditional and unconditional GANs. Nonetheless, recent work has also adapted adversarial losses to plenty discriminative models (e.g., image-to-image translation [@isola2017] and image super-resolution [@ledig17cvpr]). Therefore, it is worth investigating the behaviors of adversarial losses in different scenarios.

In addition, our theoretical analysis provides a new perspective on adversarial losses and reveals a large class of component functions valid for adversarial losses. We note that @nowozin2016 has also shown a certain class of component functions can result in theoretically valid adversarial losses. However, in their formulations, the component functions $f$ and $g$ are not independent of each other as they considered only the *f*-divergences. A future direction is to investigate the necessary and sufficient conditions for the existence and the uniqueness of a Nash equilibrium.

Proofs of the Theorems {#app:sec:proofs}
======================

If [Property \[prop:weak\]]{} holds, then for any $\gamma \in [0, 1]$, $\psi(\gamma) + \psi(1 - \gamma) \geq 2\,\psi(\frac{1}{2})$. \[app:theo:weak\_necessary\_condition\]

Since [Property \[prop:weak\]]{} holds, we have for any fixed $p_d$, $$\label{app:eq:theo1_prerequisite}
    L_G \geq L_G\,\big\rvert_{\,p_g = p_d}\,.$$

Let us consider $$\begin{aligned}
    p_d({\mathbf{x}}) = \gamma\,\delta({\mathbf{x}}- \mathbf{s}) + (1 - \gamma)\,\delta({\mathbf{x}}- \mathbf{t})\,,\\
    p_g({\mathbf{x}}) = (1 - \gamma)\,\delta({\mathbf{x}}- \mathbf{s}) + \gamma\,\delta({\mathbf{x}}- \mathbf{t})\,.
  \end{aligned}$$ for some $\gamma \in [0, 1]$ and $\mathbf{s}, \mathbf{t} \in {\mathcal{X}}, \mathbf{s} \neq \mathbf{t}$. Then, we have $$\begin{aligned}
    &L_G\,\big\rvert_{\,p_g = p_d}\\
    &= \max_{D}\;\int_{\mathbf{x}}p_d({\mathbf{x}})\,f(D({\mathbf{x}})) + p_d({\mathbf{x}})\,g(D({\mathbf{x}}))\,d{\mathbf{x}}\\
    &= \max_{D}\;\int_{\mathbf{x}}p_d({\mathbf{x}})\,(f(D({\mathbf{x}})) + g(D({\mathbf{x}})))\,d{\mathbf{x}}\\
    \begin{split}
      &= \max_{D}\;\int_{\mathbf{x}}\big(\,(\gamma\,\delta({\mathbf{x}}- \mathbf{s}) + (1 - \gamma)\,\delta({\mathbf{x}}- \mathbf{t}))\\
      &\qquad\qquad\qquad\qquad\qquad (f(D({\mathbf{x}})) + g(D({\mathbf{x}})))\,\big)\,d{\mathbf{x}}\end{split}\\
    \begin{split}
      \label{app:eq:theo1_note1}
      &= \max_{D}\;\big(\,\gamma (f(D(\mathbf{s})) + g(D(\mathbf{s})))\\
      &\qquad\qquad\qquad + (1 - \gamma)\,(f(D(\mathbf{t})) + g(D(\mathbf{t})))\,\big)
    \end{split}\\
    \begin{split}
      \label{app:eq:theo1_note2}
      &= \max_{y_1, y_2}\;\big(\,\gamma (f(y_1) + g(y_1))\\
      &\qquad\qquad\qquad + (1 - \gamma) (f(y_2) + g(y_2))\,\big)
    \end{split}\\
    \begin{split}
      &= \max_{y_1}\;\gamma\,(f(y_1) + g(y_1))\\
      &\qquad\qquad\qquad + \max_{y_2}\;(1 - \gamma)\,(f(y_2) + g(y_2))
    \end{split}\\
    &= \max_{y}\;f(y) + g(y)\\
    \label{app:eq:theo1_note3}
    &= 2\,\psi(\tfrac{1}{2})\,.
  \end{aligned}$$

Moreover, we have $$\begin{aligned}
    &L_G\nonumber\\
    &= \max_{D}\;\int_{\mathbf{x}}p_d({\mathbf{x}})\,f(D({\mathbf{x}})) + p_g({\mathbf{x}})\,g(D({\mathbf{x}}))\,d{\mathbf{x}}\\
    \begin{split}
      &= \max_{D}\;\int_{\mathbf{x}}\big(\,(\gamma\,\delta({\mathbf{x}}- \mathbf{s}) + (1 - \gamma)\,\delta({\mathbf{x}}- \mathbf{t}))\,f(D({\mathbf{x}}))\\
      &\;\qquad + ((1 - \gamma)\,\delta({\mathbf{x}}- \mathbf{s}) + \gamma\,\delta({\mathbf{x}}- \mathbf{t}))\,g(D({\mathbf{x}}))\,\big)\,d{\mathbf{x}}\end{split}\\
    \begin{split}
      \label{app:eq:theo1_note4}
      &= \max_{D}\;\big(\,\gamma\,f(D(\mathbf{s})) + (1 - \gamma)\,f(D(\mathbf{t}))\\
      &\qquad\qquad\qquad + (1 - \gamma)\,g(D(\mathbf{s})) + \gamma\,g(D(\mathbf{t}))\,\big)
    \end{split}\\
    \begin{split}
      \label{app:eq:theo1_note5}
      &= \max_{y_1, y_2}\;\big(\,\gamma\,f(y_1) + (1 - \gamma)\,g(y_1))\\
      &\qquad\qquad\qquad + (1 - \gamma)\,f(y_2) + \gamma\,g(y_2)\,\big)
    \end{split}\\
    \begin{split}
      &= \max_{y_1}\;\gamma\,f(y_1) + (1 - \gamma)\,g(y_1))\\
      &\qquad\qquad\qquad + \max_{y_2} (1 - \gamma)\,f(y_2) + \gamma\,g(y_2)
    \end{split}\\
    \label{app:eq:theo1_note6}
    &= \psi(\gamma) + \psi(1 - \gamma)\,.
  \end{aligned}$$ (Note that we can obtain from and from because $D$ can be any function and thus $D(\mathbf{s})$ is independent of $D(\mathbf{t})$.)

As holds for any fixed $p_d$, by substituting and into , we get $$\psi(\gamma) + \psi(1 - \gamma) \geq 2\,\psi(\tfrac{1}{2})$$ for any $\gamma \in [0, 1]$, which concludes the proof.

If [Property \[prop:strong\]]{} holds, then for any $\gamma \in [0, 1] \setminus \{\frac{1}{2}\}$, $\psi(\gamma) + \psi(1 - \gamma) > 2\,\psi(\frac{1}{2})$. \[app:theo:strong\_necessary\_condition\]

Since [Property \[prop:strong\]]{} holds, we have for any fixed $p_d$, $$\label{app:eq:theo2_prerequisite}
    L_G\big\rvert_{\,p_g \neq p_d} > L_G\,\big\rvert_{\,p_g = p_d}\,.$$

Following the proof of [Theorem \[app:theo:weak\_necessary\_condition\]]{}, consider $$\begin{aligned}
    \label{app:eq:theo2_note1}
    &p_d({\mathbf{x}}) = \gamma\,\delta({\mathbf{x}}- \mathbf{s}) + (1 - \gamma)\,\delta({\mathbf{x}}- \mathbf{t})\,,\\[1ex]
    \label{app:eq:theo2_note2}
    &p_g({\mathbf{x}}) = (1 - \gamma)\,\delta({\mathbf{x}}- \mathbf{s}) + \gamma\,\delta({\mathbf{x}}- \mathbf{t})\,,
  \end{aligned}$$ for some $\gamma \in [0, 1]$ and some $\mathbf{s}, \mathbf{t} \in {\mathcal{X}}, \mathbf{s} \neq \mathbf{t}$. It can be easily shown that $p_g = p_d$ if and only if $\gamma = \tfrac{1}{2}$.

As holds for any fixed $p_d$, by substituting and into , we get $$\psi(\gamma) + \psi(1 - \gamma) > 2\,\psi(\tfrac{1}{2})\,,$$ for any $\gamma \in [0, 1] \setminus \{\tfrac{1}{2}\}$, concluding the proof.

If $\psi(\gamma)$ has a global minimum at $\gamma = \frac{1}{2}$, then [Property \[prop:weak\]]{} holds. \[app:theo:weak\_sufficient\_condition\]

First, we see that $$\begin{aligned}
    &L_G\,\big\rvert_{\,p_g = p_d}\\
    &= \max_{D}\;\int_{\mathbf{x}}p_d({\mathbf{x}})\,f(D({\mathbf{x}})) + p_d({\mathbf{x}})\,g(D({\mathbf{x}}))\,d{\mathbf{x}}\\
    &= \max_{y}\;\int_{\mathbf{x}}p_d({\mathbf{x}})\,f(y) + p_d({\mathbf{x}})\,g(y)\,d{\mathbf{x}}\\
    &= \max_{y}\;\int_{\mathbf{x}}p_d({\mathbf{x}})\,(f(y) + g(y))\,d{\mathbf{x}}\\
    &= \max_{y}\;(f(y) + g(y)) \int_{\mathbf{x}}p_d({\mathbf{x}})\,d{\mathbf{x}}\\
    &= \max_{y}\;f(y) + g(y)\\
    \label{app:eq:theo3_note1}
    &= 2\,\psi(\tfrac{1}{2})\,.
  \end{aligned}$$ On the other had, we have $$\begin{aligned}
    &L_G\nonumber\\
    &= \max_{D}\;\int_{\mathbf{x}}p_d({\mathbf{x}})\,f(D({\mathbf{x}})) + p_g({\mathbf{x}})\,g(D({\mathbf{x}}))\,d{\mathbf{x}}\\
    &= \max_{y}\;\int_{\mathbf{x}}p_d({\mathbf{x}})\,f(y) + p_g({\mathbf{x}})\,g(y)\,d{\mathbf{x}}\\
    \begin{split}
      &= \max_{y}\;\int_{\mathbf{x}}(p_d({\mathbf{x}}) + p_g({\mathbf{x}}))\\
      &\qquad\qquad \left(\frac{p_d({\mathbf{x}})\,f(y)}{p_d({\mathbf{x}}) + p_g({\mathbf{x}})} + \frac{p_g({\mathbf{x}})\,g(y)}{p_d({\mathbf{x}}) + p_g({\mathbf{x}})}\right)\,d{\mathbf{x}}\end{split}\\
    \begin{split}
      &= \int_{\mathbf{x}}(p_d({\mathbf{x}}) + p_g({\mathbf{x}}))\\
      &\qquad\qquad \max_{y} \left(\frac{p_d({\mathbf{x}})\,f(y)}{p_d({\mathbf{x}}) + p_g({\mathbf{x}})} + \frac{p_g({\mathbf{x}})\,g(y)}{p_d({\mathbf{x}}) + p_g({\mathbf{x}})}\right)\,d{\mathbf{x}}\,.
    \end{split}
  \end{aligned}$$ Since $\frac{p_d({\mathbf{x}})}{p_d({\mathbf{x}}) + p_g({\mathbf{x}})} \in [0, 1]$, we have $$\label{app:eq:theo3_note2}
    L_G = \int_{\mathbf{x}}(p_d({\mathbf{x}}) + p_g({\mathbf{x}}))\,\psi\left(\frac{p_d({\mathbf{x}})}{p_d({\mathbf{x}}) + p_g({\mathbf{x}})}\right)\,d{\mathbf{x}}\,.$$ As $\psi(\gamma)$ has a global minimum at $\gamma = \frac{1}{2}$, now we have $$\begin{aligned}
    L_G &\geq \int_{\mathbf{x}}(p_d({\mathbf{x}}) + p_g({\mathbf{x}}))\,\psi(\tfrac{1}{2})\,d{\mathbf{x}}\\
    &= \psi(\tfrac{1}{2}) \int_{\mathbf{x}}(p_d({\mathbf{x}}) + p_g({\mathbf{x}}))\,d{\mathbf{x}}\\
    \label{app:eq:theo3_note3}
    &= 2\,\psi(\tfrac{1}{2})\,.
  \end{aligned}$$ Finally, combining and yields $$L_G \geq L_G\,\big\rvert_{\,p_g = p_d}\,,$$ which holds for any $p_d$, thus concluding the proof.

If $\psi(\gamma)$ has a unique global minimum at $\gamma = \frac{1}{2}$, then [Property \[prop:strong\]]{} holds. \[app:theo:strong\_sufficient\_condition\]

Since $\psi(\gamma)$ has a unique global minimum at $\gamma = \frac{1}{2}$, we have for any $\gamma \in [0, 1] \setminus \frac{1}{2}$, $$\psi(\gamma) > \psi(\tfrac{1}{2})\,.$$ When $p_g \neq p_d$, there must be some ${\mathbf{x}}_0 \in {\mathcal{X}}$ such that $p_g({\mathbf{x}}_0) \neq p_d({\mathbf{x}}_0)$. Thus, $\frac{p_d({\mathbf{x}}_0)}{p_d({\mathbf{x}}_0) + p_g({\mathbf{x}}_0)} \neq \frac{1}{2}$, and thereby $\psi\left(\frac{p_d({\mathbf{x}}_0)}{p_d({\mathbf{x}}_0) + p_g({\mathbf{x}}_0)}\right) > \psi(\frac{1}{2})$. Now, by we have $$\begin{aligned}
    &L_G\,\big\rvert_{\,p_g \neq p_d}\\
    &= \int_{\mathbf{x}}(p_d({\mathbf{x}}) + p_g({\mathbf{x}}))\,\psi\left(\frac{p_d({\mathbf{x}})}{p_d({\mathbf{x}}) + p_g({\mathbf{x}})}\right)\,d{\mathbf{x}}\\
    &> \int_{\mathbf{x}}(p_d({\mathbf{x}}) + p_g({\mathbf{x}}))\,\psi(\tfrac{1}{2})\,d{\mathbf{x}}\\
    &= \psi(\tfrac{1}{2}) \int_{\mathbf{x}}(p_d({\mathbf{x}}) + p_g({\mathbf{x}}))\,d{\mathbf{x}}\\
    \label{app:eq:theo4_note1}
    &= 2\,\psi(\tfrac{1}{2})\,.
  \end{aligned}$$ Finally, combining and yields $$L_G\,\big\rvert_{\,p_g \neq p_d} > L_G\,\big\rvert_{\,p_g = p_d}\,,$$ which holds for any $p_d$, thus concluding the proof.

If $f'' + g'' \leq 0$ and there exists some $y^*$ such that $f(y^*) = g(y^*)$ and $f'(y^*) = -g'(y^*) \neq 0$, then $\psi(\gamma)$ has a unique global minimum at $\gamma = \frac{1}{2}$. \[app:theo:strong\_sufficient\_condition2\]

First, we have by definition $$\Psi(\gamma, y) = \gamma\,f(y) + (1 - \gamma)\,g(y)\,.$$ By taking the partial derivatives, we get $$\begin{aligned}
    \label{app:eq:theo5_partial1}
    &\frac{\partial\Psi}{\partial\gamma} = f(y) - g(y)\,,\\[1ex]
    \label{app:eq:theo5_partial2}
    &\frac{\partial\Psi}{\partial y} = \gamma\,f'(y) + (1 - \gamma)\,g'(y)\,,\\[1ex]
    \label{app:eq:theo5_partial3}
    &\frac{\partial^2\Psi}{\partial y^2} = \gamma\,f''(y) + (1 - \gamma)\,g''(y)\,.
  \end{aligned}$$

We know that there exists some $y^*$ such that $$\begin{aligned}
    \label{app:eq:theo5_prerequisite1}
    &f(y^*) = g(y^*)\,,\\[1ex]
    \label{app:eq:theo5_prerequisite2}
    &f'(y^*) = -g'(y^*) \neq 0\,.
  \end{aligned}$$

1.  By and , we see that $$\begin{aligned}
            \label{app:eq:theo5_note1}
            &\frac{\partial\Psi}{\partial\gamma}\,\Big\rvert_{\,y = y^*} = 0\,,\\[1ex]
            \label{app:eq:theo5_note2}
            &\frac{\partial\Psi}{\partial y}\,\Big\rvert_{\,(\gamma, y) = (\frac{1}{2}, y^*)} = 0\,.
          \end{aligned}$$ Now, by we know that $\Psi$ is constant when $y = y^*$. That is, for any $\gamma \in [0, 1]$, $$\Psi(\gamma, y^*) = \Psi(\tfrac{1}{2}, y^*)\,.$$

2.  Because $f'' + g'' \leq 0$, by we have $$\begin{aligned}
            \label{app:eq:theo5_note3}
            \frac{\partial^2\Psi}{\partial y^2}\,\Big\rvert_{\,\gamma = \tfrac{1}{2}} &= \tfrac{1}{2}\,f''(y) + \tfrac{1}{2}\,g''(y)\\
            &\leq 0\,.
          \end{aligned}$$ By and , we see that $y^*$ is a global minimum point of $\Psi\big\rvert_{\gamma = \tfrac{1}{2}}$. Thus, we now have $$\begin{aligned}
            \Psi(\tfrac{1}{2}, y^*) &= \max_y\;\Psi(\tfrac{1}{2}, y)\\
            \label{app:eq:theo5_note4}
            &= \psi(\tfrac{1}{2})\,.
          \end{aligned}$$

3.  By , we see that $$\begin{aligned}
            \frac{\partial\Psi}{\partial y}\,\Big\rvert_{\,y = y^*}
            &= \gamma\,f'(y^*) + (1 - \gamma)\,g'(y^*)\\
            &= \gamma\,f'(y^*) + (1 - \gamma)\,(-f'(y^*))\\
            &= (2 \gamma - 1)\,f'(y^*)\,.
          \end{aligned}$$ Since $f'(y^*) \neq 0$, we have $$\frac{\partial\Psi}{\partial y}\,\Big\rvert_{\,y = y^*} \neq 0\quad\forall\,\gamma \in [0, 1] \setminus \tfrac{1}{2}\,.$$ This shows that for any $\gamma \in [0, 1] \setminus \tfrac{1}{2}$, there must exists some $y^\circ$ such that $$\label{app:eq:theo5_note5}
            \Psi(\gamma, y^\circ) > \Psi(\gamma, y^*)\,.$$ And by definition we have $$\begin{aligned}
            \label{app:eq:theo5_note6}
            \Psi(\gamma, y^\circ) &< \max_y\;\Psi(\gamma, y)\\
            &= \psi(\gamma)\,.
          \end{aligned}$$ Hence, by and we get $$\label{app:eq:theo5_note7}
            \psi(\gamma) > \Psi(\gamma, y^*)\,.$$

Finally, combining , and yields $$\psi(\gamma) > \psi(\tfrac{1}{2})\quad\forall\,\gamma \in [0, 1] \setminus \tfrac{1}{2}\,,$$ which concludes the proof.

More Graphs of the $\Psi$ and $\psi$ Functions {#app:sec:psi_function_graphs}
==============================================

We show in [ \[app:fig:small\_psi\_functions\]]{} the graphs of the $\psi$ functions for different adversarial losses. Note that for the Wasserstein loss, the $\psi$ function is only defined at $\gamma = 0.5$, where it takes the value of zero, and for the asymmetric loss, the $\psi$ function is only defined when $\gamma > 0.5$, where it takes the value of zero. Hence, we do not include them in [ \[app:fig:small\_psi\_functions\]]{}.

We also present in [ \[app:fig:psi\_functions\_imbalanced\]]{} the graphs of the $\Psi$ functions for the $\epsilon$-weighted versions of the classic, the Wasserstein and the hinge losses. Moreover, Figures \[app:fig:small\_psi\_functions\](b) and (c) show the graphs of the $\psi$ functions for the $\epsilon$-weighted versions of the classic and the hinge losses, respectively.

Network Architectures {#app:sec:net_architectures}
=====================

We present in [ \[app:tab:network\_architectures\]]{} the network architectures for the generator and the discriminator used for all the experiments.

More Results {#app:sec:results}
============

We report in [ \[app:tab:exp\_loss\_functions\]]{} the results for the one-side coupled and local gradient penalties.

We also present in [ \[app:fig:exp\_momentum\]]{} the results for the experiment on the momentum terms using the hinge loss.

\
(a) common adversarial losses\

  ------------------------------------------------------- -----------------------------------------------------
                                                          
   \(b) $\epsilon$-weighted versions of the classic loss   \(c) $\epsilon$-weighted versions of the hinge loss
  ------------------------------------------------------- -----------------------------------------------------

[lccc]{}\
*conv* &32 &3$\times$3 &3$\times$3\
*conv* &64 &3$\times$3 &3$\times$3\
*maxpool* &- &2$\times$2 &2$\times$2\
*dense* &128\
*dense* &10\

[lccc]{}\
*conv* &32 &3$\times$3 &3$\times$3\
*conv* &64 &3$\times$3 &3$\times$3\
*maxpool* &- &2$\times$2 &2$\times$2\
*dense* &128\
*dense* &1\

                                                         OCGP                OLGP              SN + OCGP           SN + OLGP
  ----------------------------------------------- ------------------- ------------------- ------------------- -------------------
  classic (M) [@goodfellow2014]                      7.15$\pm$0.77     **6.95$\pm$0.51**     7.16$\pm$0.31       6.86$\pm$0.29
  classic (N) [@goodfellow2014]                      7.20$\pm$0.39       6.98$\pm$0.22       7.47$\pm$0.62       7.15$\pm$0.36
  classic (L)                                        7.12$\pm$0.61       7.00$\pm$1.00       7.29$\pm$0.35       7.18$\pm$0.54
  hinge (M)                                        **5.82$\pm$0.31**     7.33$\pm$1.35     **5.80$\pm$0.24**     5.83$\pm$0.20
  hinge (N)                                                              7.88$\pm$1.33       5.92$\pm$0.36     **5.74$\pm$0.27**
  hinge (L) [@lim2017; @tran2017]                  **5.77$\pm$0.29**   **6.22$\pm$1.04**   **5.77$\pm$0.30**   **5.82$\pm$0.20**
  Wasserstein [@arjovsky2017wgan]                    7.60$\pm$3.02      13.34$\pm$1.49       6.35$\pm$0.43       6.06$\pm$0.45
  least squares [@mao2017]                           7.99$\pm$0.35       8.06$\pm$0.49       8.43$\pm$0.50       8.31$\pm$0.52
  relativistic [@jolicoeur-martineau2018]            8.03$\pm$3.32       9.41$\pm$2.90       6.18$\pm$0.29       6.03$\pm$0.24
  relativistic hinge [@jolicoeur-martineau2018]     10.70$\pm$2.51      14.17$\pm$1.79                        
  absolute                                           5.95$\pm$0.19                           6.22$\pm$0.25       6.08$\pm$0.32
  asymmetric                                         5.85$\pm$0.35       7.57$\pm$0.98       6.21$\pm$0.34       5.92$\pm$0.37

[^1]: This is possibly because the R~1~ and the R~2~ gradient penalties encourage $D$ to have small gradients, and thus the gradients for both $D$ and $G$ might vanish when $p_g$ and $p_d$ are close enough.

[^2]: A possible reason is that as $p_g$ move towards $p_d$, the gradients for $G$ become smaller (and eventually zero when $p_d = p_g$), which can slow down the training. The two-side penalties can alleviate this by encouraging the norm of the gradients to be a fixed value.
