A Generative Model for Hallucinating Diverse Versions of\\ Super Resolution Images
Traditionally, the main focus of image super-resolution techniques is on recovering the most likely high-quality images from low-quality images, using a one-to-one low- to high-resolution mapping. Proceeding that way, we ignore the fact that there are generally many valid versions of high-resolution images that map to a given low-resolution image. We are tackling in this work the problem of obtaining different high-resolution versions from the same low-resolution image using Generative Adversarial Models. Our learning approach makes use of high frequencies available in the training high-resolution images for preserving and exploring in an unsupervised manner the structural information available within these images. Experimental results on the CelebA dataset confirm the effectiveness of the proposed method, which allows the generation of both realistic and diverse high-resolution images from low-resolution images.
Template.tex
% Template for ICIP-2018 paper; to be used with:
%          spconf.sty  - ICASSP/ICIP LaTeX style file, and
%          IEEEbib.bst - IEEE bibliography style file.
% --------------------------------------------------------------------------
\documentclass{article}
\usepackage{spconf,amsmath,graphicx}
\usepackage{amsfonts}
\usepackage{float}
\usepackage{booktabs}

% Example definitions.
% --------------------
\def\x{{\mathbf x}}
\def\L{{\cal L}}

% Title.
% ------
\title{A Generative Model for Hallucinating Diverse Versions of\\ Super Resolution Images}
%
% Single address.
% ---------------
\name{Mohamed Abid, Ihsen Hedhli, Christian Gagné$^*$ }
\address{IID, Université Laval, $^*$Canada-CIFAR AI Chair, Mila}
% For example:
% ------------
%\address{School\\
% Department\\
% Address}
%
% Two addresses (uncomment and modify for two-address case).
% ----------------------------------------------------------
%\twoauthors
%  {A. Author-one, B. Author-two\sthanks{Thanks to XYZ agency for funding.}}
% {School A-B\\
% Department A-B\\
% Address A-B}
%  {C. Author-three, D. Author-four\sthanks{The fourth author performed the work
% while at ...}}
% {School C-D\\
% Department C-D\\
% Address C-D}
%
\begin{document}
%\ninept
%
\maketitle
%
\begin{abstract}
Traditionally, the main focus of image super-resolution techniques is on recovering the most likely high-quality images from low-quality images, using a one-to-one low- to high-resolution mapping. Proceeding that way, we ignore the fact that there are generally many valid versions of high-resolution images that map to a given low-resolution image. We are tackling in this work the problem of obtaining different high-resolution versions from the same low-resolution image using Generative Adversarial Models. Our learning approach makes use of high frequencies available in the training high-resolution images for preserving and exploring in an unsupervised manner the structural information available within these images. Experimental results on the CelebA dataset confirm the effectiveness of the proposed method, which allows the generation of both realistic and diverse high-resolution images from low-resolution images.
\end{abstract}
%
\begin{keywords}
Super-resolution, Image Hallucination, Generative Adversarial Models.
\end{keywords}
%
\section{Introduction}
\label{sec:intro}

Image super-resolution (SR) is an ill-posed problem, since each low-resolution (LR) image can have a practically infinite number of corresponding images in the high-resolution (HR) domain. However, traditional SR techniques rely on a one-to-one schema, aiming at generating (or reconstructing) only one HR image from a LR image, generally the most likely one. However, they still do not account for all other possible mappings. In practice, in applications such as microscopy or medical imaging, where experts rely on SR methods, having access to more than one possible solution can lead to better strategies where different outcomes, and even the certainty on these outcomes, are considered.

In this paper, we seek to perform a one-to-many schema by generating a diverse set of HR images out of one single LR image. For that, we are proposing to use a Generative Adversarial Network (GAN) that recovers an HR image from a very LR version of the same image, meanwhile being able to hallucinate a wide variety of other acceptable HR versions of this image. In the recent literature, some exploratory SR has been investigated, for example relying on SR methods that are either controlled by a user interface to manually manipulate the variance and periodicity of textures \cite{bahat}, or by using semantically guided style imposition \cite{deepsee}. The diversity generated in these methods are supervised, by either a human or a specific criterion, which is in contrast to our method where hallucination is conducted in an unsupervised way. Another approach such as \cite{pulse} searches in a pretrained GAN’s manifold for a set of high-resolution images that maps, when downscaled, to the same low-resolution image. While this method yields great quality samples, it is an iterative procedure that requires a significant amount of time to generate each image. Meanwhile our method is a one pass-prediction which works for general images and is not dependent on a pretrained GAN (e.g., StyleGAN). 
\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{fig1.pdf}
    \caption{Conceptual representation of proposed approach: starting from an original image ($I^\mathit{HR}$) and its downscaled version ($I^\mathit{LR}$), different HR images can be generated, that is here the reconstructed image ($I^\mathit{SR}$) and two hallucinated ones ($I^\mathit{H}_{z_1}$ and $I^\mathit{H}_{z_2}$).}
    \label{fig:concept}
\end{figure}

Methods such as \cite{prevgrad} exploited the image’s gradient as structure guidance for super-resolution. This helps the model to better learn and preserve structural and textural information in the image. We follow a similar path but we are not limiting its use to preserve the structure of the image but to anchor the exploration over the image-gradient manifold in order to generate other images that would preserve most of the structure given by the image’s gradient. The main contribution of the paper lies in the novel neural architecture that makes use of high-frequency within the image not only to recover one HR image but also to generate a wide variety of plausible HR images that downscale to the same LR version of of this image -- see Fig.~\ref{fig:concept} that illustrates the concept of our approach.


\section{Approach}
\label{sec:format}

%%Given a single low-resolution image, the objective of the proposed approach is twofold. First, both the High-resolution image and its corresponding gradient one are reconstructed. Second, a variety of high resolution images that down-scale to the input low-resolution image are hallucinated.
Given a LR image $I^\mathit{LR}$, the objective of the proposed approach is twofold:
\begin{itemize} 
    \item Reconstruct the ground truth HR image that allows the retrieval of its corresponding image gradient;
    \item Hallucinate a variety of HR images that match, when downscaled, the input LR image ($I^\mathit{LR})$. 
\end{itemize}
To tackle the aforementioned objectives, we propose a generator $G$ that takes as input pairs of $(I^\mathit{LR},\,z)$, where  $z$ is a $m$-dimensional white noise vector, $z \sim \mathcal{N}^m(0,1)$. 
We follow the same assumptions made in \cite{lag}, with our proposed model deemed to reconstruct the ground-truth HR image when $z=0$, otherwise ($z \neq 0$) sampling a variety of images matching the LR input but having a varied image gradient:
\begin{align}
G(I^\mathit{LR},\,0) & = (I^\mathit{SR},\,g^\mathit{SR})  \hspace{2.25em}\text{(reconstruction)},\label{eq:1}\\
G(I^\mathit{LR},\,z) & = (I^\mathit{H}_{z},\,g^{H}_{z})  \hspace{3em}\text{(hallucination)}.\label{eq:2}
\end{align}
In these equations, $I^\mathit{SR}$ is the reconstructed ground-truth image while $I^\mathit{H}_{z}$ is a hallucinated version given a noise vector $z$. Also, $g^\mathit{SR}$ and $g^\mathit{H}_{z}$ are respectively the reconstructed image gradient of the ground truth image and the hallucinated image gradient given a noise vector $z$.

\subsection{Image Reconstruction}

Like traditional SR models, the proposed model aims to minimize the distance between the ground-truth and the reconstructed HR images. In addition, our model aims to preserve structural information in the original image by using the ground-truth image $I^\mathit{HR}$ and its correspond gradient $g^\mathit{HR}$, computed as follows:
\begin{align}
g_{x} &=I^\mathit{HR}(x+1,\,y)-I^\mathit{HR}(x-1,\,y),\\
g_{y} &=I^\mathit{HR}(x,\,y+1)-I^\mathit{HR}(x,\,y-1),\\
g^\mathit{HR} &= \|(g_{x},\,g_{y})\|_2.
\end{align}
As shown in \eqref{eq:1}, the generator takes as input the pair $(I^\mathit{LR},\,0)$ and outputs $(I^\mathit{SR},\,g^{SR})$. The Perceptual loss \cite{perceploss} with VGG-16 \cite{Vgg} is then used by comparing the obtained $I^\mathit{SR}$ to the real $ I^\mathit{HR}$, by minimizing the Euclidean distances between the features of these images: 
\begin{equation}
\mathcal{L}_\mathit{percp}=\mathbb{E}_{I^\mathit{SR}}\left\|\phi_{i}(I^\mathit{SR})-\phi_{i}(I^\mathit{HR})\right\|_{1},\label{per}
\end{equation}
where $\phi_i$ is value output by the $i$-th layer of VGG.
As for comparing the gradient images, the mean absolute error of the images (pixels) is used:
\begin{equation}
\mathcal{L}_{grad}=\left\|g^\mathit{SR} - g^\mathit{HR}\right\|_{1}.\label{gradrecon}
\end{equation}
Finally, for both domains, we use an adversarial loss for assessing the quality of the images and gradients generated, that is a non-saturating logistic loss \cite{goodfellow} with $R_1$ regularization \cite{r1}.

\subsection{Image Hallucination}
Unlike the reconstruction, image hallucination does not rely on comparing the generated image to the ground truth image. Now in addition to the low-resolution input image $I^\mathit{LR}$, the proposed method uses a random vector $z$ to generate and explore other plausible solutions.  
However, it is worth noting that conditional GANs (cGANs) suffer from mode collapse problems \cite{sal,mode}. Also cGANs tend to ignore the random vector $z$ when conditioned on inputs that contain significant information about the output, for example in image-to-image translation models \cite{pix2pix, cyclegan} where the model tends to ignore the random vector. Without addressing this problem in cGANs, the hallucinated images would be equal to the reconstructed image and the model will regress to the same behaviour of one-to-one mapping. To circumvent this problem, we impose on the model a diversity constraint first introduced in \cite{diversity}, which is applied here only on the gradient output, to orient exploration of the structural and texture space of the gradient image.

To enforce the effect of $z$ for obtaining diverse solutions, the generator aims at \emph{maximizing} loss $\mathcal{L}_{z}(G)$: 
\begin{equation}
\mathcal{L}_{z}(G)=\mathbb{E}_{z_{1},z_{2} \sim \mathcal{N}^n(0,1)}\left[\min \left(\frac{d(g^{H}_{z_1},\,g^{H}_{z_2})}{\|z_{1}-z_{2}\|}, \tau\right)\right],
\end{equation}
where $d(\cdot,\cdot)$ is a distance metric, and $g^\mathit{H}_{z_1}$ and $g^\mathit{H}_{z_2}$ are the gradient images obtained from HR hallucinated images with $z_1$ and $z_2$ sampled from $\mathcal{N}^n(0,1)$ , respectively, both hallucinated images obtained from the same LR input. The minimum with $\tau$ is conducted to ensure numerical stability of the optimization, in case where the output is not bounded by an activation function (e.g., tanh or sigmoid). 

Maximizing $\mathcal{L}_{z}(G)$ encourages the generator to explore more the HR space, by producing a varied set of hallucinated samples. To make sure that the hallucinated samples stay faithful to the given LR input image, we impose the following constraint:
\begin{equation}
\|\mathrm{DS}(I^{H}_{z}) - I^{LR}\| < \epsilon,\label{eq:epsi}
\end{equation}
where $\mathrm{DS}(\cdot)$ is a down-scaling operation, and $\epsilon$ is a hyper-parameter.

\subsection{Overall Objective}

Two discriminators are used in the generative model, one for assessing the HR images ($D_{I}$) and the other to validate its corresponding gradient ($D_{g}$). For both networks, non-saturating logistic loss \cite{goodfellow} with $R_1$ regularization \cite{r1} is used:
%\begin{align}
%\mathcal{L}_{adv}^{g}=&-\mathbb{E}_{g^{HR}}\left[\log \left(1-D_{g}\left(g^{HR}\right)\right)\right] \nonumber \\
%&-\mathbb{E}_{g^{SR}}\left[\log D_{g}\left(g^{SR}\right)\right]  \label{eq:323}
%\end{align}
\begin{align*}
\mathcal{L}_{adv}^{g} &= -\mathbb{E}_{g^\mathit{HR}}\left[\log(1-D_{g}(g^\mathit{HR}))\right] - \mathbb{E}_{g^\mathit{SR}}\left[\log D_{g}(g^\mathit{SR})\right],\\
\mathcal{L}_{adv}^{I} &= -\mathbb{E}_{I^\mathit{HR}}\left[\log(1-D_{I}(I^\mathit{HR}))\right] - \mathbb{E}_{I^\mathit{SR}}\left[\log D_{I}(I^{SR})\right].  
\end{align*}

%Following \eqref{eq:323}, \eqref{eq:585} :
%\begin{equation}
%\mathcal{L}_{adv} = \mathcal{L}_{adv}^{I} + \mathcal{L}_{adv}^{g} \label{eq:adv}
%\end{equation}

As for generating reconstructed HR images, we use a weighted sum over $\mathcal{L}_\mathit{percp}$ \eqref{per}, $\mathcal{L}_\mathit{grad}$ \eqref{gradrecon} and the adversarial loss:
\begin{equation}
\mathcal{L}_\textit{recons} = \gamma(\mathcal{L}_\mathit{percp} + \mathcal{L}_\mathit{grad})
     + \beta (\mathcal{L}_{adv}^{g} + \mathcal{L}_{adv}^{I}).
\end{equation}
And for generating hallucinated HR images, the loss is:
\begin{equation}
    \mathcal{L}_\textit{halluc} = \mathcal{L}_{adv}^{g} + \mathcal{L}_{adv}^{I} + \alpha \mathcal{L}_{z}.
\end{equation}

\section{Implementation}

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{fig2.pdf}
    \caption{Generator Architecture: the network consists of 8 residual blocks followed by stacked upsampling blocks (Upblocks) -- see Fig.~\ref{fig:upblock} for the inner layers of UpBlock.}
    \label{fig:gen}
\end{figure}

\subsection{Conditioning}

We choose to condition the discriminator $D_I$ on the constraint \eqref{eq:epsi} by concatenating it at the last block of the network.
We follow \cite{lag} by rounding the downscaled fake image to the closest colour value and dividing it by $r = 2/255$ (since images are in the range $[-1,1]$).
\begin{equation}
F = \max\left( \frac{\left|\lfloor DS(I^\mathit{H}_z) \rceil - I^\mathit{LR}\right|}{r} - \epsilon,\,0\right).\label{eq:f}
\end{equation}
That prevents producing exceedingly large values for the discriminator’s weights in order to measure infinitesimal differences.
A straight through estimator is used to pass the backpropagation gradient through the rounding operation.
Therefore, the discriminator receives two inputs, fake/real images and $F$ given in \eqref{eq:f}:
$$
\left\{\begin{array}{ll}
D_I\left(I^\mathit{HR},\,0\right) & \text{for real images} \\
D_I\left(I^\mathit{H}_{z},\,F\right) & \text{for fake images} 
\end{array}\right..
$$

\subsection{Architecture}

Our model consists of a generator, that takes a LR image, a noise vector, and outputs two images: a HR image and its corresponding gradients. Two discriminators are also used during the learning, which are taking real/fake images and their gradients, respectively.

The generator global architecture is inspired by \cite{MSGGAN} (see Fig.~\ref{fig:gen}). First the residual blocks take the LR image with noise vectors, then they are followed by up-sampling blocks, where each block outputs an image and its corresponding gradient at that scale (see Fig.~\ref{fig:upblock}). This way the gradient flows to the generator at different scales, allowing it to be more stable, leading to a faster learning. 
\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{fig3.pdf}
    \caption{Upblock architecture: a convolution block shared between the two domains is used to enable the learning of image structures, which are then followed by two domain-specific convolution and a $1\times 1$ convolution to make a projection of the values in the feature space into the pixel space. The key in using a skip connection is to keep the low frequency information such as colours intact.}
    \label{fig:upblock}
\end{figure}
As mentioned before, two discriminators are used, one for each domain ($I^\mathit{HR}$ and $g^\mathit{HR}$), their architecture is mirrored version of the generator, but since each discriminator is domain-specific, the upblock is replaced by a downblock, where we use first a $1\times 1$ convolution to pass the image from the pixel space to feature space. This is followed by another two convolutions, by leaky RELUs and then by average pooling for downsampling. After the downsampling blocks, the residual blocks are just like the generator. The discriminator directly outputs the features of last convolutions, without using linear layers.

%\subsection{UpBlock}
%\label{upblock}
%Each upblock use a shared convolution between the two domains, then followed by two domain-specific convolution and 1 × 1 convolution to project the features into the pixel space.

%The key in sharing the first convolution is to focus on learning structural in formation in the image, however, to keep the information about colours and other textural information intact, we use a skip connection that only pass to the image domain but not to the gradient domain.  


%\begin{figure*}[h!]
%   \includegraphics[scale=0.62]{viz.png}
%    \caption{Real image and its Gradient, and Hallucinated Images and its corresponding Gradient}
%    \label{fig:grad}
%\end{figure*}

\begin{figure*}[t]
\centering
\includegraphics[width=0.78\linewidth]{drawing.pdf}
\caption{Results on 8x scaling: reconstruction (SR) and hallucination using different $z$ vectors}
\label{fig:viz}
\end{figure*}

\section{Experiments}

\subsection{Settings}
Experiments were conducted on the CelebA \cite{celeba} dataset, to assess the capability of our model, which was trained on upscaling images 8 times and then compared to other SR methods (i.e, \cite{srgan} and \cite{esrgan}). 

Our model is trained with the Adam optimizer\cite{adam} ($lr = 1e^{-4}$, $beta_1= 0.$, $beta_2= 0.9$) with TTUR \cite{ttur}, using batch sizes equal to 8. We use a leaky RELU for all activation functions in the generator and both discriminators, with a leak equal to 0.2.

We fix $\epsilon$ in \eqref{eq:f} to $0.1$, while for $\mathcal{L}_\mathit{recons}$ we set $\gamma = 10$ and $\beta=0.1$, and for $\mathcal{L}_\mathit{halluc}$, $\alpha $ is equal to $1$.


\begin{figure}[t!]
    \centering
    \includegraphics[width=0.95\linewidth]{fig4.pdf}
    \caption{Results on CelebA 8x scaling: (a) ground truth, (b) bicubic reconstruction, (c) SRGAN, (d) Esrgan, (e) Ours : Reconstruction ($z=0$), and (f) Ours : Hallucination ($z \neq 0$).}
    \label{fig:im}
\end{figure}
\begin{table}[tb!]
\centering
\begin{tabular}{@{}lccc@{}}
\toprule
Method  & SSIM $\uparrow$ & PSNR $\uparrow$ & LPIPS $\downarrow$\\ \midrule
Bicubic & 0.43 & 18.11 & 0.51  \\
SRGAN \cite{srgan}   & 0.62 & 20.15 & 0.24  \\
ESRGAN \cite{esrgan}  & 0.55 & 18.43 & 0.23  \\
Ours    & \textbf{0.68} & \textbf{21.08} &\textbf{0.18}  \\ \bottomrule
\end{tabular}
\caption{Comparison of results for 8x scaling on CelebA \cite{celeba}, all results are reported on the test set.}
\label{tab:results}
\end{table}

\subsection{Results}
\label{sec:pagestyle}

We use the traditional SR metrics peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) and learned perceptual image patch similarity (LPIPS) \cite{nossim}. Although our method is optimized to generate images with high perceptual quality, which means that the results do not correlate very well with PSNR and SSIM \cite {nossim}, we still achieve excellent performance, with an improvement by a large margin on all metrics -- see Table~\ref{tab:results} for details.

%\begin{figure}
%    \centering
%    \includegraphics[scale=0.4]{grad.pdf}
%    \caption{Real image and its Gradient, and Hallucinated Images and its corresponding Gradient}
%    \label{fig:grad}
%\end{figure}

We report the results of our model and compare these results with the results obtained using the approaches described in \cite{srgan,esrgan}.
We show that when the input is a very low resolution image, where most of the information is missing, the use of the gradient has a huge impact on the results.
As illustrated in Fig.~\ref{fig:im}, the reconstructions of our model are closer to the real images than other methods, but also more coherent since the structural properties of the images is being preserved by making the model learn the gradient of the image.

In addition to having better reconstruction given the use of $\epsilon = 0.1$ for \eqref{eq:epsi} in our experiments, the model has more freedom to hallucinate other semantics that do not appear in the low-resolution image, such as changing the facial expressions or even changing the person's identity (see Fig.\ \ref{fig:viz}).
%The generator associates the latent space $z$ to the semantics of the face, to the textural details in the hair. (see Figure \ref{fig:viz})

%The generator is able to learn to generate both the image and its corresponding gradient %(see Figure \ref{fig:grad}), using only one shared backbone not like other methods that use two separated generators each for a specific domain \cite{prevgrad}.


\section{Conclusion}
In this paper we propose a new super-resolution method that makes use of novel GAN  architecture that is able to recover images from extremely low resolution images, and hallucinate a wide variety of other possibilities.  We showed the superiority of our approach in both tasks on the CelebA dataset.

%\section{Acknowledgement}
%This research paper is funded by Thales Canada.
\newpage
% References should be produced using the bibtex program from suitable
% BiBTeX files (here: strings, refs, manuals). The IEEEbib.bst bibliography
% style file from IEEE produces unsorted bibliography list.
% -------------------------------------------------------------------------
\bibliographystyle{IEEEbib}
\bibliography{strings,refs}

\end{document}


Template.bbl
\begin{thebibliography}{10}

\bibitem{bahat}
Yuval {Bahat} and Tomer {Michaeli},
\newblock ``{Explorable Super Resolution},''
\newblock {\em arXiv e-prints}, p. arXiv:1912.01839, Dec. 2019.

\bibitem{deepsee}
Marcel~Christoph B{\"u}hler, A.~Romero, and R.~Timofte,
\newblock ``Deepsee: Deep disentangled semantic explorative extreme
  super-resolution,''
\newblock {\em ArXiv}, vol. abs/2004.04433, 2020.

\bibitem{pulse}
Sachit {Menon}, Alexandru {Damian}, Shijia {Hu}, Nikhil {Ravi}, and Cynthia
  {Rudin},
\newblock ``{PULSE: Self-Supervised Photo Upsampling via Latent Space
  Exploration of Generative Models},''
\newblock {\em arXiv e-prints}, p. arXiv:2003.03808, Mar. 2020.

\bibitem{prevgrad}
Cheng {Ma}, Yongming {Rao}, Yean {Cheng}, Ce~{Chen}, Jiwen {Lu}, and Jie
  {Zhou},
\newblock ``{Structure-Preserving Super Resolution with Gradient Guidance},''
\newblock {\em arXiv e-prints}, p. arXiv:2003.13081, Mar. 2020.

\bibitem{lag}
David {Berthelot}, Peyman {Milanfar}, and Ian {Goodfellow},
\newblock ``{Creating High Resolution Images with a Latent Adversarial
  Generator},''
\newblock {\em arXiv e-prints}, p. arXiv:2003.02365, Mar. 2020.

\bibitem{perceploss}
Justin Johnson, Alexandre Alahi, and Fei{-}Fei Li,
\newblock ``Perceptual losses for real-time style transfer and
  super-resolution,''
\newblock {\em CoRR}, vol. abs/1603.08155, 2016.

\bibitem{Vgg}
Karen Simonyan and Andrew Zisserman,
\newblock ``Very deep convolutional networks for large-scale image
  recognition,''
\newblock in {\em 3rd International Conference on Learning Representations,
  {ICLR} 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track
  Proceedings}, Yoshua Bengio and Yann LeCun, Eds., 2015.

\bibitem{goodfellow}
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
  Sherjil Ozair, Aaron Courville, and Yoshua Bengio,
\newblock ``Generative adversarial nets,''
\newblock in {\em Advances in Neural Information Processing Systems},
  Z.~Ghahramani, M.~Welling, C.~Cortes, N.~Lawrence, and K.~Q. Weinberger, Eds.
  2014, vol.~27, pp. 2672--2680, Curran Associates, Inc.

\bibitem{r1}
Lars~M. Mescheder,
\newblock ``On the convergence properties of {GAN} training,''
\newblock {\em CoRR}, vol. abs/1801.04406, 2018.

\bibitem{sal}
Tim Salimans, Ian~J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford,
  and Xi~Chen,
\newblock ``Improved techniques for training gans,''
\newblock {\em CoRR}, vol. abs/1606.03498, 2016.

\bibitem{mode}
Martin {Arjovsky} and L{\'e}on {Bottou},
\newblock ``{Towards Principled Methods for Training Generative Adversarial
  Networks},''
\newblock {\em arXiv e-prints}, p. arXiv:1701.04862, Jan. 2017.

\bibitem{pix2pix}
Phillip Isola, Jun{-}Yan Zhu, Tinghui Zhou, and Alexei~A. Efros,
\newblock ``Image-to-image translation with conditional adversarial networks,''
\newblock {\em CoRR}, vol. abs/1611.07004, 2016.

\bibitem{cyclegan}
Jun{-}Yan Zhu, Taesung Park, Phillip Isola, and Alexei~A. Efros,
\newblock ``Unpaired image-to-image translation using cycle-consistent
  adversarial networks,''
\newblock {\em CoRR}, vol. abs/1703.10593, 2017.

\bibitem{diversity}
Dingdong Yang, Seunghoon Hong, Yunseok Jang, Tianchen Zhao, and Honglak Lee,
\newblock ``Diversity-sensitive conditional generative adversarial networks,''
\newblock {\em CoRR}, vol. abs/1901.09024, 2019.

\bibitem{MSGGAN}
Animesh Karnewar, Oliver Wang, and Raghu~Sesha Iyengar,
\newblock ``{MSG-GAN:} multi-scale gradient {GAN} for stable image synthesis,''
\newblock {\em CoRR}, vol. abs/1903.06048, 2019.

\bibitem{celeba}
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang,
\newblock ``Deep learning face attributes in the wild,''
\newblock in {\em Proceedings of International Conference on Computer Vision
  (ICCV)}, December 2015.

\bibitem{srgan}
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew~P. Aitken,
  Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi,
\newblock ``Photo-realistic single image super-resolution using a generative
  adversarial network,''
\newblock {\em CoRR}, vol. abs/1609.04802, 2016.

\bibitem{esrgan}
Xintao Wang, Ke~Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Chen~Change
  Loy, Yu~Qiao, and Xiaoou Tang,
\newblock ``{ESRGAN:} enhanced super-resolution generative adversarial
  networks,''
\newblock {\em CoRR}, vol. abs/1809.00219, 2018.

\bibitem{adam}
Diederik~P. Kingma and Jimmy Ba,
\newblock ``Adam: {A} method for stochastic optimization,''
\newblock in {\em 3rd International Conference on Learning Representations,
  {ICLR} 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track
  Proceedings}, Yoshua Bengio and Yann LeCun, Eds., 2015.

\bibitem{ttur}
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler,
  G{\"{u}}nter Klambauer, and Sepp Hochreiter,
\newblock ``Gans trained by a two time-scale update rule converge to a nash
  equilibrium,''
\newblock {\em CoRR}, vol. abs/1706.08500, 2017.

\bibitem{nossim}
Richard Zhang, Phillip Isola, Alexei~A. Efros, Eli Shechtman, and Oliver Wang,
\newblock ``The unreasonable effectiveness of deep features as a perceptual
  metric,''
\newblock {\em CoRR}, vol. abs/1801.03924, 2018.

\end{thebibliography}

refs.bib

@article{ttur,
  author    = {Martin Heusel and
               Hubert Ramsauer and
               Thomas Unterthiner and
               Bernhard Nessler and
               G{\"{u}}nter Klambauer and
               Sepp Hochreiter},
  title     = {GANs Trained by a Two Time-Scale Update Rule Converge to a Nash Equilibrium},
  journal   = {CoRR},
  volume    = {abs/1706.08500},
  year      = {2017},
  url       = {http://arxiv.org/abs/1706.08500},
  archivePrefix = {arXiv},
  eprint    = {1706.08500},
  timestamp = {Sat, 23 Jan 2021 01:20:58 +0100},
  biburl    = {https://dblp.org/rec/journals/corr/HeuselRUNKH17.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
@inproceedings{adam,
  author    = {Diederik P. Kingma and
               Jimmy Ba},
  editor    = {Yoshua Bengio and
               Yann LeCun},
  title     = {Adam: {A} Method for Stochastic Optimization},
  booktitle = {3rd International Conference on Learning Representations, {ICLR} 2015,
               San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings},
  year      = {2015},
  url       = {http://arxiv.org/abs/1412.6980},
  timestamp = {Thu, 25 Jul 2019 14:25:37 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/KingmaB14.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{nossim,
  author    = {Richard Zhang and
               Phillip Isola and
               Alexei A. Efros and
               Eli Shechtman and
               Oliver Wang},
  title     = {The Unreasonable Effectiveness of Deep Features as a Perceptual Metric},
  journal   = {CoRR},
  volume    = {abs/1801.03924},
  year      = {2018},
  url       = {http://arxiv.org/abs/1801.03924},
  archivePrefix = {arXiv},
  eprint    = {1801.03924},
  timestamp = {Wed, 14 Aug 2019 08:23:33 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1801-03924.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
@inproceedings{celeba,
 title = {Deep Learning Face Attributes in the Wild},
 author = {Liu, Ziwei and Luo, Ping and Wang, Xiaogang and Tang, Xiaoou},
 booktitle = {Proceedings of International Conference on Computer Vision (ICCV)},
 month = {December},
 year = {2015} 
}
@article{srgan,
  author    = {Christian Ledig and
               Lucas Theis and
               Ferenc Huszar and
               Jose Caballero and
               Andrew P. Aitken and
               Alykhan Tejani and
               Johannes Totz and
               Zehan Wang and
               Wenzhe Shi},
  title     = {Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial
               Network},
  journal   = {CoRR},
  volume    = {abs/1609.04802},
  year      = {2016},
  url       = {http://arxiv.org/abs/1609.04802},
  archivePrefix = {arXiv},
  eprint    = {1609.04802},
  timestamp = {Mon, 13 Aug 2018 16:48:38 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/LedigTHCATTWS16.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

@article{esrgan,
  author    = {Xintao Wang and
               Ke Yu and
               Shixiang Wu and
               Jinjin Gu and
               Yihao Liu and
               Chao Dong and
               Chen Change Loy and
               Yu Qiao and
               Xiaoou Tang},
  title     = {{ESRGAN:} Enhanced Super-Resolution Generative Adversarial Networks},
  journal   = {CoRR},
  volume    = {abs/1809.00219},
  year      = {2018},
  url       = {http://arxiv.org/abs/1809.00219},
  archivePrefix = {arXiv},
  eprint    = {1809.00219},
  timestamp = {Sat, 23 Jan 2021 01:19:59 +0100},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1809-00219.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

@article{MSGGAN,
  author    = {Animesh Karnewar and
               Oliver Wang and
               Raghu Sesha Iyengar},
  title     = {{MSG-GAN:} Multi-Scale Gradient {GAN} for Stable Image Synthesis},
  journal   = {CoRR},
  volume    = {abs/1903.06048},
  year      = {2019},
  url       = {http://arxiv.org/abs/1903.06048},
  archivePrefix = {arXiv},
  eprint    = {1903.06048},
  timestamp = {Sun, 31 Mar 2019 19:01:24 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1903-06048.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

@article{bahat,
       author = {{Bahat}, Yuval and {Michaeli}, Tomer},
        title = "{Explorable Super Resolution}",
      journal = {arXiv e-prints},
     keywords = {Computer Science - Computer Vision and Pattern Recognition},
         year = 2019,
        month = dec,
          eid = {arXiv:1912.01839},
        pages = {arXiv:1912.01839},
archivePrefix = {arXiv},
       eprint = {1912.01839},
 primaryClass = {cs.CV},
       adsurl = {https://ui.adsabs.harvard.edu/abs/2019arXiv191201839B},
      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

@article{deepsee,
  title={DeepSEE: Deep Disentangled Semantic Explorative Extreme Super-Resolution},
  author={Marcel Christoph B{\"u}hler and A. Romero and R. Timofte},
  journal={ArXiv},
  year={2020},
  volume={abs/2004.04433}
}


@ARTICLE{pulse,
       author = {{Menon}, Sachit and {Damian}, Alexandru and {Hu}, Shijia and {Ravi}, Nikhil and {Rudin}, Cynthia},
        title = "{PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models}",
      journal = {arXiv e-prints},
     keywords = {Computer Science - Computer Vision and Pattern Recognition, Computer Science - Machine Learning, Electrical Engineering and Systems Science - Image and Video Processing},
         year = 2020,
        month = mar,
          eid = {arXiv:2003.03808},
        pages = {arXiv:2003.03808},
archivePrefix = {arXiv},
       eprint = {2003.03808},
 primaryClass = {cs.CV},
       adsurl = {https://ui.adsabs.harvard.edu/abs/2020arXiv200303808M},
      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}



@article{sal,
  author    = {Tim Salimans and
               Ian J. Goodfellow and
               Wojciech Zaremba and
               Vicki Cheung and
               Alec Radford and
               Xi Chen},
  title     = {Improved Techniques for Training GANs},
  journal   = {CoRR},
  volume    = {abs/1606.03498},
  year      = {2016},
  url       = {http://arxiv.org/abs/1606.03498},
  archivePrefix = {arXiv},
  eprint    = {1606.03498},
  timestamp = {Sun, 16 Sep 2018 14:17:04 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/SalimansGZCRC16.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}



@ARTICLE{mode,
       author = {{Arjovsky}, Martin and {Bottou}, L{\'e}on},
        title = "{Towards Principled Methods for Training Generative Adversarial Networks}",
      journal = {arXiv e-prints},
     keywords = {Statistics - Machine Learning, Computer Science - Machine Learning},
         year = 2017,
        month = jan,
          eid = {arXiv:1701.04862},
        pages = {arXiv:1701.04862},
archivePrefix = {arXiv},
       eprint = {1701.04862},
 primaryClass = {stat.ML},
       adsurl = {https://ui.adsabs.harvard.edu/abs/2017arXiv170104862A},
      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}



@article{pix2pix,
  author    = {Phillip Isola and
               Jun{-}Yan Zhu and
               Tinghui Zhou and
               Alexei A. Efros},
  title     = {Image-to-Image Translation with Conditional Adversarial Networks},
  journal   = {CoRR},
  volume    = {abs/1611.07004},
  year      = {2016},
  url       = {http://arxiv.org/abs/1611.07004},
  archivePrefix = {arXiv},
  eprint    = {1611.07004},
  timestamp = {Mon, 13 Aug 2018 16:49:05 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/IsolaZZE16.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

@article{diversity,
  author    = {Dingdong Yang and
               Seunghoon Hong and
               Yunseok Jang and
               Tianchen Zhao and
               Honglak Lee},
  title     = {Diversity-Sensitive Conditional Generative Adversarial Networks},
  journal   = {CoRR},
  volume    = {abs/1901.09024},
  year      = {2019},
  url       = {http://arxiv.org/abs/1901.09024},
  archivePrefix = {arXiv},
  eprint    = {1901.09024},
  timestamp = {Sat, 02 Feb 2019 16:56:00 +0100},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1901-09024.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

@article{cyclegan,
  author    = {Jun{-}Yan Zhu and
               Taesung Park and
               Phillip Isola and
               Alexei A. Efros},
  title     = {Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial
               Networks},
  journal   = {CoRR},
  volume    = {abs/1703.10593},
  year      = {2017},
  url       = {http://arxiv.org/abs/1703.10593},
  archivePrefix = {arXiv},
  eprint    = {1703.10593},
  timestamp = {Mon, 13 Aug 2018 16:48:06 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/ZhuPIE17.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}


@ARTICLE{prevgrad,
       author = {{Ma}, Cheng and {Rao}, Yongming and {Cheng}, Yean and {Chen}, Ce and {Lu}, Jiwen and {Zhou}, Jie},
        title = "{Structure-Preserving Super Resolution with Gradient Guidance}",
      journal = {arXiv e-prints},
     keywords = {Electrical Engineering and Systems Science - Image and Video Processing, Computer Science - Computer Vision and Pattern Recognition},
         year = 2020,
        month = mar,
          eid = {arXiv:2003.13081},
        pages = {arXiv:2003.13081},
archivePrefix = {arXiv},
       eprint = {2003.13081},
 primaryClass = {eess.IV},
       adsurl = {https://ui.adsabs.harvard.edu/abs/2020arXiv200313081M},
      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

@inproceedings{Vgg,
  author    = {Karen Simonyan and
               Andrew Zisserman},
  editor    = {Yoshua Bengio and
               Yann LeCun},
  title     = {Very Deep Convolutional Networks for Large-Scale Image Recognition},
  booktitle = {3rd International Conference on Learning Representations, {ICLR} 2015,
               San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings},
  year      = {2015},
  url       = {http://arxiv.org/abs/1409.1556},
  timestamp = {Wed, 17 Jul 2019 10:40:54 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/SimonyanZ14a.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
@article{perceploss,
  author    = {Justin Johnson and
               Alexandre Alahi and
               Fei{-}Fei Li},
  title     = {Perceptual Losses for Real-Time Style Transfer and Super-Resolution},
  journal   = {CoRR},
  volume    = {abs/1603.08155},
  year      = {2016},
  url       = {http://arxiv.org/abs/1603.08155},
  archivePrefix = {arXiv},
  eprint    = {1603.08155},
  timestamp = {Sat, 19 Oct 2019 16:30:04 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/JohnsonAL16.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
@ARTICLE{lag,
       author = {{Berthelot}, David and {Milanfar}, Peyman and {Goodfellow}, Ian},
        title = "{Creating High Resolution Images with a Latent Adversarial Generator}",
      journal = {arXiv e-prints},
     keywords = {Computer Science - Computer Vision and Pattern Recognition, Electrical Engineering and Systems Science - Image and Video Processing, Statistics - Machine Learning},
         year = 2020,
        month = mar,
          eid = {arXiv:2003.02365},
        pages = {arXiv:2003.02365},
archivePrefix = {arXiv},
       eprint = {2003.02365},
 primaryClass = {cs.CV},
       adsurl = {https://ui.adsabs.harvard.edu/abs/2020arXiv200302365B},
      adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

@article{r1,
  author    = {Lars M. Mescheder},
  title     = {On the convergence properties of {GAN} training},
  journal   = {CoRR},
  volume    = {abs/1801.04406},
  year      = {2018},
  url       = {http://arxiv.org/abs/1801.04406},
  archivePrefix = {arXiv},
  eprint    = {1801.04406},
  timestamp = {Mon, 13 Aug 2018 16:46:00 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-1801-04406.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
@inproceedings{goodfellow,
 author = {Goodfellow, Ian and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and Warde-Farley, David and Ozair, Sherjil and Courville, Aaron and Bengio, Yoshua},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {Z. Ghahramani and M. Welling and C. Cortes and N. Lawrence and K. Q. Weinberger},
 pages = {2672--2680},
 publisher = {Curran Associates, Inc.},
 title = {Generative Adversarial Nets},
 url = {https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf},
 volume = {27},
 year = {2014}
}

strings.bib
@Article{Lamp86,
  author =       "A.B. Smith and C.D. Jones and E.F. Roberts",
  title =        "Article Title",
  journal = 	 "Journal",
  year = 	 "1920",
  volume = 	 "62",
  pages = 	 "291-294",
  month = 	 "January"
}

