  Recent years have seen adversarial losses been applied to many fields. Their
applications extend beyond the originally proposed generative modeling to
conditional generative and discriminative settings. While prior work has
proposed various output activation functions and regularization approaches,
some open questions still remain unanswered. In this paper, we aim to study the
following two research questions: 1) What types of output activation functions
form a well-behaved adversarial loss? 2) How different combinations of output
activation functions and regularization approaches perform empirically against
one another? To answer the first question, we adopt the perspective of
variational divergence minimization and consider an adversarial loss
well-behaved if it behaves as a divergence-like measure between the data and
model distributions. Using a generalized formulation for adversarial losses, we
derive the necessary and sufficient conditions of a well-behaved adversarial
loss. Our analysis reveals a large class of theoretically valid adversarial
losses. For the second question, we propose a simple comparative framework for
adversarial losses using discriminative adversarial networks. The proposed
framework allows us to efficiently evaluate adversarial losses using a standard
evaluation metric such as the classification accuracy. With the proposed
framework, we evaluate a comprehensive set of 168 combinations of twelve output
activation functions and fourteen regularization approaches on the handwritten
digit classification problem to decouple their effects. Our empirical findings
suggest that there is no single winning combination of output activation
functions and regularization approaches across all settings. Our theoretical
and empirical results may together serve as a reference for choosing or
designing adversarial losses in future research.
