Learning Super-Resolution Jointly from External and Internal Examples
\authorblockN{Zhangyang Wang
Yingzhe
Qing Ling and Weiping Li
Zhangyang Wang\dag
Yingzhen Yang\dag
Zhaowen Wang\ddag
Shiyu Chang\dag
Jianchao Yang\ddag
Thomas S. Huang\dag
\dag Beckman Institute
University of Illinois at Urbana-Champaign
Urbana
IL 61801
USA
\ddag Adobe Systems Inc
San Jose
CA 95110
USA
{\tt\small \{zwang119
yyang58\
Zhangyang~Wang
% Houqiang~Li
~\IEEEmembership{Member
~IEEE
%\normalsize Single image super-resolution (SR) aims to estimate a high-resolution (HR) image from a low-resolution (LR) input. Image priors are commonly learned to regularize the otherwise seriously ill-posed SR problem, either using external LR-HR pairs or internal similar patterns. We propose joint SR to adaptively combine the advantages of both external and internal SR methods. We define two loss functions using sparse coding based external examples, and epitomic matching based on internal examples, as well as a corresponding adaptive weight to automatically balance their contributions according to their reconstruction errors. Extensive SR results demonstrate the effectiveness of the proposed method over the existing state-of-the-art methods, and is also verified by our subjective evaluation study.
epitome-symbol.tex
\newcommand{\ve}{\mathbf{e}}
\newcommand{\vemean}{\mathbf{\mu}}
\newcommand{\vevar}{\mathbf{\phi}}
\newcommand{\vpi}{\bm{\pi}}


\newcommand{\semean}{\bm{\mu}}
\newcommand{\sevar}{\bm{\phi}}

\newcommand{\vX}{\mathbf{X}}
\newcommand{\vY}{\mathbf{Y}}
\newcommand{\vZ}{\mathbf{Z}}
\newcommand{\vS}{\mathbf{S}}
\newcommand{\vT}{\mathcal{T}}
\newcommand{\vN}{\mathcal{N}}

\newcommand{\vI}{\mathcal{I}}
\newcommand{\vx}{\mathbf{x}}
JSR_double.tex
%\documentclass[10pt,journal,cspaper,compsoc]{IEEEtran}

\documentclass[10pt,twocolumn,twoside]{IEEEtran}
%\documentclass[11pt,draftcls,onecolumn]{IEEEtran}
\usepackage{times}
\usepackage{graphicx}
\usepackage{subfigure}
\usepackage{amsmath}
\usepackage{tabularx}
\usepackage{amssymb}
\usepackage{mathrsfs}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{multirow}
\usepackage{bm}
\usepackage{url}
\input{epitome-symbol.tex}

%\newcommand{\remove}[1]{{\color{Gray}#1}}
%\newcommand{\comm}[1]{{\color{red}#1}}
%\newcommand{\revise}[1]{{\color{blue}#1}}

\begin{document}
\title{Learning Super-Resolution Jointly from External and Internal Examples}

%\author{\authorblockN{Zhangyang Wang, Yingzhe, Qing Ling and Weiping Li\\}
%\authorblockA{Dept. of EEIS, University of Science and Technology of China, Hefei, P.R. China\\
%wzy302@mail.ustc.edu.cn, lihq@ustc.edu.cn,
%qingling@mail.ustc.edu.cn, wpli@ustc.edu.cn} }

\author{Zhangyang Wang\dag, Yingzhen Yang\dag, Zhaowen Wang\ddag, Shiyu Chang\dag, Jianchao Yang\ddag, Thomas S. Huang\dag
 \\
\dag Beckman Institute, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA\\
\ddag Adobe Systems Inc, San Jose, CA 95110, USA\\
{\tt\small \{zwang119, yyang58\}@illinois.edu, zhawang@adobe.com,  chang87@illinois.edu, jiayang@adobe.com, t-huang1@illinois.edu}
% For a paper whose authors are all at the same institution,
% omit the following lines up until the closing ``}''.
% Additional authors and addresses can be added with ``\and'',
% just like the second author.
% To save space, use either the email address or home page, not both
}



%\author{Zhangyang~Wang,
%        Houqiang~Li,~\IEEEmembership{Member,~IEEE},
%        Qing~Ling
%        and~Weiping~Li,~\IEEEmembership{Fellow,~IEEE}% <-this % stops a space
%\IEEEcompsocitemizethanks{\IEEEcompsocthanksitem The authors are with the School of Information Science and Technology, University of Science and Technology of China, Hefei, P.R. China.\protect\\
%% note need leading \protect in front of \\ to get a newline within \thanks as
%% \\ is fragile and will error, could use \hfil\break instead.
%Professor Houqiang Li is the corresponding author. His e-mail: lihq@ustc.edu.cn.}
%}


%% two-column
%\IEEEcompsocitemizethanks{\IEEEcompsocthanksitem Manuscript received Nov 11, 2011. (Write the date on which you submitted your paper for review.)\protect\\
%\IEEEcompsocthanksitem Z. Wang is with the Electrical Engineering and Information Science Department, University of Science and Technology of China, Hefei, 230027 China (e-mail: wzy302@mail.ustc.edu.cn).\protect\\
%\IEEEcompsocthanksitem H. Li is with the Electrical Engineering and Information Science Department, University of Science and Technology of China, Hefei, 230027 China (corresponding author, phone: 86-551-3601340; e-mail: lihq@ ustc.edu.cn).\protect\\
%\IEEEcompsocthanksitem Q. Ling is with the Automation Department, University of Science and Technology of China, Hefei, 230027 China (e-mail: qingling@mail.ustc.edu.cn).\protect\\
%\IEEEcompsocthanksitem W. Li is with the Electrical Engineering and Information Science Department, University of Science and Technology of China, Hefei, 230027 China (e-mail: wpli@ ustc.edu.cn).\protect\\
%% note need leading \protect in front of \\ to get a newline within \thanks as
%% \\ is fragile and will error, could use \hfil\break instead.
%}
%}

% The paper headers
%\markboth{SUBMITTED TO IEEE TRANSACTIONS ON IMAGE PROCESSING}%
%{Shell \MakeLowercase{\textit{et al.}}: Bare Demo of IEEEtran.cls for Computer Society Journals}



\IEEEcompsoctitleabstractindextext{%
\begin{abstract}
%\normalsize
Single image super-resolution (SR) aims to estimate a high-resolution (HR) image from a low-resolution (LR) input. Image priors are commonly learned to regularize the otherwise seriously ill-posed SR problem, either using external LR-HR pairs or internal similar patterns. We propose joint SR to adaptively combine the advantages of both external and internal SR methods. We define two loss functions using sparse coding based external examples, and epitomic matching based on internal examples, as well as a corresponding adaptive weight to automatically balance their contributions according to their reconstruction errors. Extensive SR results demonstrate the effectiveness of the proposed method over the existing state-of-the-art methods, and is also verified by our subjective evaluation study.
\end{abstract}

\begin{keywords}
%\normalsize
Super-resolution, example-based methods, sparse coding, epitome\\

\textbf{EDICS} Category: TEC-ISR Interpolation, Super-Resolution, and Mosaicing
\end{keywords}}

\maketitle

%\newpage

\IEEEdisplaynotcompsoctitleabstractindextext
\IEEEpeerreviewmaketitle

%\newpage

\section{Introduction}


Super-resolution (SR) algorithms aim to constructing a high-resolution (HR) image from one or multiple low-resolution (LR) input frames \cite{Park2003}. This problem is essentially ill-posed because much information is lost in the HR to LR degradation process. Thus SR has to refer to strong image priors, that range from the simplest analytical smoothness assumptions, to more sophisticated statistical and structural priors learned from natural images \cite{Fattal2007}, \cite{Lin2004}, \cite{Yang2012}, \cite{Fattal2010}. The most popular single image SR methods rely on example-based learning techniques. Classical example-based methods learn the mapping between LR and HR image patches, from a large and representative external set of image pairs, and is thus denoted as \textit{external SR}. Meanwhile, images generally possess a great amount of self-similarities; such a self-similarity property motivates a series of \textit{internal SR} methods. With much progress being made, it is recognized that external and internal SR methods each suffer from their certain drawbacks. However, their complementary properties inspire us to propose the \textit{joint super-resolution} (joint SR),  that adaptively utilizes both external and internal examples for the SR task. The contributions of this paper are multi-fold:
\begin{itemize}

\item We propose joint SR exploiting both external and internal examples, by defining an adaptive combination of different loss functions.

\item We apply \textit{epitomic matching} \cite{JojicFK03} to enforcing self-similarity in SR. Compared the the local nearest neighbor (NN) matching adopted in \cite{Fattal2010}, epitomic matching features more robustness to outlier features, as well as the ability to perform efficient non-local searching.

\item We carry out a human subjective evaluation survey to evaluate SR result quality based on visual perception, among several state-of-the-art methods. 

%Experimental results confirm the superiority of the new method in both quantitative and qualitative comparisons.
% [the formulation is general, but the solution is specific. This may weaken our contributions.]

\end{itemize}

%The remainder of the thesis is organized as follows. Section 2 first studies the motivation and related literature for joint SR. We then develop our joint SR model and discuss each of its components in Section 3. Section 4 presents implementation details and results of our algorithm, with comparison to a few state-of-the-art methods. Finally, we conclude the thesis in Section 5.


\section{A Motivation Study of Joint SR}

\begin{figure*}[htbp]
\centering
\begin{minipage}{0.24\textwidth}
\centering \subfigure[\textit{Train}, the groundtruth of \newline carriage region] {
\includegraphics[width=\textwidth]{cropGT.jpg}
}\end{minipage}
\begin{minipage}{0.24\textwidth}
\centering \subfigure[\textit{Train}, carriage region by \cite{Yang2012} \newline PSNR = 24.91 dB, SSIM = 0.7915] {
\includegraphics[width=\textwidth]{crop.jpg}
}\end{minipage}
\begin{minipage}{0.24\textwidth}
\centering \subfigure [ \textit{Train}, carriage region by \cite{Fattal2010} \newline PSNR = 24.13 dB, SSIM = 0.8085] {
\includegraphics[width=\textwidth]{crop2.jpg}
}\end{minipage}
\\
\begin{minipage}{0.24\textwidth}
\centering \subfigure[\textit{Train}, the groundtruth of \newline brick region] {
\includegraphics[width=\textwidth]{jsrcrop_GT.jpg}
}\end{minipage}
\begin{minipage}{0.24\textwidth}
\centering \subfigure [\textit{Train}, brick region by \cite{Yang2012} \newline PSNR = 18.84 dB, SSIM = 0.6576] {
\includegraphics[width=\textwidth]{jsr_1-crop.jpg}
}\end{minipage}
\begin{minipage}{0.24\textwidth}
\centering \subfigure [\textit{Train}, brick region by \cite{Fattal2010} \newline PSNR = 19.78 dB, SSIM = 0.7037] {
\includegraphics[width=\textwidth]{jsr_1-crop2.jpg}
}\end{minipage}
\\
\begin{minipage}{0.24\textwidth}
\centering \subfigure[\textit{Kid}, the groundtruth of  \newline left eye region] {
\includegraphics[width=\textwidth]{1GT.jpg}
}\end{minipage}
\begin{minipage}{0.24\textwidth}
\centering \subfigure[\textit{Kid}, left eye region by \cite{Yang2012} \newline PSNR = 22.43 dB, SSIM = 0.6286] {
\includegraphics[width=\textwidth]{1b.jpg}
}\end{minipage}
\begin{minipage}{0.24\textwidth}
\centering \subfigure [\textit{Kid}, left eye region by \cite{Fattal2010} \newline PSNR = 22.18 dB, SSIM = 0.5993] {
\includegraphics[width=\textwidth]{1a.jpg}
}\end{minipage}
\\
\begin{minipage}{0.24\textwidth}
\centering \subfigure [\textit{Kid}, the groundtruth of \newline sweater region] {
\includegraphics[width=\textwidth]{2GT.jpg}
}\end{minipage}
\begin{minipage}{0.24\textwidth}
\centering \subfigure [\textit{Kid}, sweater region by \cite{Yang2012} \newline PSNR = 24.16 dB, SSIM = 0.5444] {
\includegraphics[width=\textwidth]{2b.jpg}
}\end{minipage}
\begin{minipage}{0.24\textwidth}
\centering \subfigure [ \textit{Kid}, sweater region by \cite{Fattal2010} \newline PSNR = 24.45 dB, SSIM = 0.6018] {
\includegraphics[width=\textwidth]{2a.jpg}
}\end{minipage}
\caption{Visual comparisons of both external and internal SR methods on different image local regions. The PSNR and SSIM values are also calculated and reported.}
\label{fig:1}
\end{figure*}


\subsection{Related Work}
The joint utilization of both external and internal examples has been most studied for image denoising \cite{Zontak2011}. Mosseri et. al. \cite{Irani} first proposed that some image patches inherently prefer internal examples for denoising, whereas other patches inherently prefer external denoising. Such a preference is in essence the tradeoff between noise-fitting versus signal-fitting. Burger et. al. \cite{Harold} proposed a learning-based approach that automatically combines denoising results from an internal and an external method. The learned combining strategy outperforms both internal and external approaches across a wide range of images, being closer to theoretical bounds.

In SR literature, while the most popular methods are based on either external or internal similarities, there have been limited efforts to utilize one to regularize the other. The authors in \cite{Dong} incorporated both a local autoregressive (AR) model and a nonlocal self-similarity regularization term, into the sparse representation framework, weighted by constant coefficients. Yang et. al. \cite{Yang2013} learned the (approximated) nonlinear SR mapping function from a collection of external images with the help of in-place self-similarity. More recently, an explicitly joint model is put forward in \cite{WACV}, including two loss functions by sparse coding and local scale invariance, bound by an indicator function to decide which loss function will work for each patch of the input image. Despite the existing efforts, there is little understanding on how the external and internal examples interact with each other in SR, how to judge the external versus internal preference for each patch, and how to make them collaborate towards an overall optimized performance. 

External SR methods use a universal set of example patches to predict the missing (high-frequency) information for the HR image. In \cite{NN},  during the training phase, LR-HR patch pairs are collected. Then in the test phase, each input LR patch is found with a nearest neighbor (NN) match in the LR patch pool, and its corresponding HR patch is selected as the output. It is further formulated as a kernel ridge regression (KRR) in \cite{RG}. More recently, a popular class of external SR methods are associated with the \textit{sparse coding} technique \cite{Lee2007}, \cite{Yang2010}. The patches of a natural image can be represented as a sparse linear combination of elements within a redundant pre-trained dictionary. Following this principle, the advanced \textit{coupled sparse coding} is further proposed in  \cite{Yang2012}, \cite{Yang2010}. External SR methods are known for their capabilities to produce plausible image appearances. However, there is no guarantee that an arbitrary input patch can be well matched or represented by the external dataset of limited size. When dealing with some unique features that rarely appear in the given dataset, external SR methods are prone to produce either noise or oversmoothness \cite{Huang2010}. It constitutes the inherent problem of any external SR method with a finite-size training set \cite{Dong2012}. 

Internal SR methods search for example patches from the input image itself, based on the fact that patches often tend to recur within the image \cite{Glasner2009}, \cite{Mairal2009}, \cite{Huang2010}, or across different image scales \cite{Fattal2010}. Although internal examples provide a limited number of references, they are very relevant to the input image. However, this type of approach has a limited performance, especially for irregular patches without any discernible repeating pattern \cite{DD}. Due to the insufficient patch pairs, the mismatches of internal examples often lead to more visual artifacts. In addition, epitome was proposed in \cite{JojicFK03, Ni09, ChuYLCH10} to summarize both local and non-local similar patches and reduces the artifacts caused by neighborhood matching. We apply epitome as an internal SR technique in this paper, and evidence its advantages by our experiments.

%For example, the method proposed in \cite{Dong} incorporated both a local autoregressive (AR) model and a nonlocal self-similarity regularization term, into the sparse representation framework. The local self-similarity is further refined by Yang et. al. \cite{Yang2013} using in-place self-similarity, meaning that patch matching can be restricted to its original location in the lower-scale image. In order to handle these in-place internal examples, the authors learned a robust first-order approximation of the nonlinear mapping function from a collection of external images. The algorithm can produce more natural structures and is thereby better at handling real applications.

\subsection{Comparing External and Internal SR Methods}

Both external and internal SR methods have different advantages and drawbacks. See Fig.~\ref{fig:1} for a few specific examples. The first two rows of images are cropped from the 3$\times$ SR results of the \textit{Train} image, and the last two rows from the 4$\times$ SR results of the \textit{Kid} image. Each row of images are cropped from the same spatial location of the groundtruth image, the SR result by the external method \cite{Yang2012}, and the SR result by the internal method \cite{Fattal2010}, respectively. In the first row, the top contour of carriage (c) contains noticeable structural deformations, and the numbers ``425'' are more blurred than those in (b). That is because the numbers can more easily find counterparts or similar structure components from an external dataset; but within the same image, there are few recurring patterns that look visually similar to the numbers. Internal examples generate sharper SR results in images (f) than (e), since the bricks repeat their own patterns frequently, and thus the local neighborhood is rich in internal examples. Another winning case of external examples is between (h) and (i), as in the latter, inconsistent artifacts along the eyelid and around the eyeball are obvious. Because the eye region is composed of complex curves and fine structures,  external examples encompass more suitable reference patches and perform a more natural-looking SR. In contrast, the repeating sweater textures lead to a sharper SR in (l) than that in (k). The PSNR and SSIM \cite{SSIM} results are also calculated for all, which further validate our visual observations. 



%The eight images form four pairs: the pair (a) (b) are cropped from the same spatial location, in the SR results of \textit{Train} by external \cite{Yang2012} and internal methods\cite{Fattal2010}, respectively (similarly for (c)(d), (e)(f), and (g)(h)). In the first pair, the top contour of carriage (b) contains noticeable structural deformations, and the numbers ``425'' are more blurred than those in (a). That is because the numbers can more easily find counterparts or similar structure components from an external dataset; but within the same image, there are few recurring patterns that look visually similar to the numbers. Internal examples generate sharper SR results in images (d) than (c), since the bricks repeat their own patterns frequently, and thus the local neighborhood is rich in internal examples. Another winning case of external examples is between (e) and (f), as inconsistent artifacts along the eyelid  and around the eyeball are obvious. Because the eye region is composed of complex curves and fine structures,  external examples encompass more suitable reference patches and perform a more natural-looking SR. In contrast, the repeating sweater textures leads to a sharper SR in (h) than that in (g). 

%The SR results of the entire \textit{Train} image are available in Section 4, and the results of \textit{Kid} image can be found in our supplementary material. 

These comparisons display the generally different, even complementary behaviors of external and internal SR. Based on the observations, we expect that the external examples contribute to visually pleasant SR results for smooth regions as well as some irregular structures that barely recur in the input. Meanwhile, internal examples serve as a powerful source to reproduce unique and singular features that rarely appear externally but repeat in the input image (or its different scales). Note that similar arguments have been validated statistically in the the image denoising literature \cite{Harold}.



%images (a) and (c) are from the 4$\times$ SR result of the \textit{Kid} image, by external SR \cite{Yang2012}, while (b) and (d) are from the 4$\times$ result of the same image but with internal SR \cite{Fattal2010}. (a) and (b) are cropped from the same spatial location, and so are images (c) and (d). First, comparing (a) with (b) shows a winning case of external examples. Although in (b), the black dot and the glare in the eye look a little sharper, it is easy to notice inconsistent artifacts along the eyelid and structural distortions on the eyeball in (b). That is because the eye region is composed of complex curves and fine structures, where few recurring patterns can be identified in the same image. In contrast, external examples encompass more suitable eye patches from an external dataset and perform a more natural-looking SR. Internal examples generate sharper SR results in images (d) than (c), since the sweater textures repeat their own patterns frequently, and thus the local neighborhood is rich in internal examples. T

%Figure~\ref{fig:1} (e) display the difference heat map between the SR error of the result by \cite{Yang2012}, and that by  \cite{Fattal2010}. The SR error is defined by the pixel-wise squared error between the SR result and the ground truth. A large value ("red") in the difference map denotes that the internal SR result is superior in squared error sense, and a small value ("green") vice versa. Although squared error is not an exact measurement to evaluate SR quality, the global complementary behavior of external and internal SR again implies the potential of their joint efforts.

%that superior SR results may be expected via a proper integration of the two. 

% between ground truth (original HR \textit{Kid} image), and SR results by bicubic interpolation, external SR and internal SR, in the form of heat map (using the ``jet'' colormap), respectively. Although both generates significantly fewer errors than the bicubic interpolation result in (g), the external SR result in (e) tends to produce fewer errors on the face, especially eye regions, whereas the internal SR result (f) shows more favorably in (repeatedly) textured regions.  



%More recently, a more explicit joint model is put forward in \cite{WACV}. The core part includes two loss functions by symmetric sparse coding and local scale invariance,  connected by a region selection function to decide which loss function will work for each patch of the input image.
\section{A Joint SR model}
Let $\mathbf{X}$ denote the HR image to be estimated from the LR input $\mathbf{Y}$. $\mathbf{X}_{ij}$ and $\mathbf{Y}_{ij}$ stand for the $(i,j)$-th ($i, j = 1, 2...$) patch from $\mathbf{X}$ and $\mathbf{Y}$, respectively.  Considering almost all SR methods work on patches, we define two loss functions $\ell_\mathcal{G} (\cdot)$ and $\ell_\mathcal{I} (\cdot)$ in a patch-wise manner, which enforce the external and internal similarities, respectively. While one intuitive idea is to minimize a weighted combination of the two loss functions, a patch-wise (adaptive) weight $\omega(\cdot) $ is needed to balance them.  We hereby write our proposed joint SR in the general form:
\begin{equation}
\begin{array}{l}\label{jsr}
\underset{\mathbf{X}_{ij}, \Theta_G, \Theta_I}{\min} \quad \ell_\mathcal{G} (\mathbf{X}_{ij}, \Theta_G|\mathbf{Y}_{ij}) + \omega(\Theta_G, \Theta_I|\mathbf{Y}_{ij}) \ell_\mathcal{I}(\mathbf{X}_{ij},  \Theta_I|\mathbf{Y}_{ij}).
\end{array}
\end{equation}
$\Theta_G$ and $\Theta_I$ are the latent representations of $\mathbf{X}_{ij}$ over the spaces of external and self examples, respectively. The form $f(\mathbf{X}_{ij}, \Theta|\mathbf{Y}_{ij})$, $f$ being $\ell_\mathcal{G}$, $\ell_\mathcal{I}$ or  $\omega$, represents the function dependent on variables $\mathbf{X}_{ij}$ and $\Theta$ ($\Theta_G$ or $\Theta_I$), with $\mathbf{Y}_{ij}$ known (we omit $\mathbf{Y}_{ij}$ in all formulations hereinafter). We will discuss each component in (\ref{jsr}) next. 

% Several extra undefined notations in (\ref{jsr}), such as $\mathbf{\alpha}_{ij}$ and $\mathbf{X}_{ij}^E$, result from the concrete choices of $\ell_\mathcal{G} (\cdot)$ and  $\ell_\mathcal{I} (\cdot)$, and will be detailed soon later.  


%The design of $\ell_\mathcal{G} \mathbf{(\cdot)}$ and $\ell_\mathcal{I} \mathbf{(\cdot)}$ can be referred to a number of prior work based on external-similarity \cite{NN}, \cite{RG}, \cite{Yang2010}, \cite{Yang2012} and self-similarity  \cite{Glasner2009}, \cite{Mairal2009}, \cite{Huang2010}.
%With $\ell_\mathcal{G} (\mathbf{X}_{ij})$ and $\ell_\mathcal{I} (\mathbf{X}_{ij})$ chosen, While one intuitive idea is to minimize a weighted combination of the two loss functions. An immediate question is then how to choose the weight and balance the two losses. We define $\omega(\mathbf{X}_{ij}) $ as the weight adaptive to current patch.

%Give the above notations, we are ready to express our joint SR model:
%\begin{equation}
%\begin{array}{l}\label{general}
%\underset{\mathbf{X}_{ij}}{\min} \quad \ell_\mathcal{G} (\mathbf{X}_{ij}) + \omega(\mathbf{X}_{ij}) \ell_\mathcal{I}(\mathbf{X}_{ij}),
%\end{array}
%\end{equation}

One specific form of joint SR will be discussed in this paper. However, note that with different choices of $\ell_\mathcal{G} \mathbf{(\cdot)}$, $\ell_\mathcal{I} \mathbf{(\cdot)}$, and $\omega(\cdot)$,  a variety of methods can be accommodated in the framework. For example, if we set $\ell_\mathcal{G} \mathbf{(\cdot)}$ as the (adaptively reweighted) sparse coding term, while choosing $\ell_\mathcal{I} \mathbf{(\cdot)}$ equivalent to the two local and non-local similarity based terms, then (\ref{jsr}) becomes the model proposed in \cite{Dong}, with $\omega(\cdot)$ being some empirically chosen constants. 

\subsection{Sparse Coding for External Examples}
The HR and LR patch spaces \{$\mathbf{X}_{ij}$\} and \{$\mathbf{Y}_{ij}$\} are assumed to be tied by some mapping function. With a well-trained coupled dictionary pair ($\mathbf{D_h}$, $\mathbf{D_l}$) (see \cite{Yang2012} for details on training a coupled dictionary pair), the \textit{coupled sparse coding} \cite{Yang2010} assumes that ($\mathbf{X}_{ij}$, $\mathbf{Y}_{ij}$) tends to admit a common sparse representation $\mathbf{a}_{ij}$. Since $\mathbf{X}$ is unknown, Yang et. al. \cite{Yang2010} suggest to first infer the sparse code $\mathbf{a}^L_{ij}$ of {$\mathbf{Y}_{ij}$} with respect to $\mathbf{D_l}$, and then use it as an approximation of $\mathbf{a}^H_{ij}$ (the sparse code of {$\mathbf{X}_{ij}$} with respect to $\mathbf{D_h}$), to recover $\mathbf{X}_{ij} \approx \mathbf{D_h} \mathbf{a}^L_{ij}$. We set $\Theta_G = \mathbf{a}_{ij}$ and constitute the loss function enforcing external similarity:
\begin{equation}
\begin{array}{l}\label{lg}
\ell_\mathcal{G} (\mathbf{X}_{ij}, \mathbf{a}_{ij}) = 
\lambda || \mathbf{a}_{ij}||_1 + || \mathbf{D_l}  \mathbf{a} _{ij}-  \mathbf{Y}_{ij}||_F^2 +  || \mathbf{D_h}  \mathbf{a}_{ij} -  \mathbf{X}_{ij}||_F^2.
\end{array}
\end{equation}

%\begin{equation}
%\begin{array}{l}\label{10}
%\underset{\mathbf{X}_{ij}, \mathbf{a}_{ij}}{\min} \quad   \lambda ||\mathbf{a}_{ij}||_1 +   ||\mathbf{D_l} \mathbf{a}_{ij} - \mathbf{Y}_{ij}||_F^2 +  ||\mathbf{D_h} \mathbf{a}_{ij} - \mathbf{X}_{ij}||_F^2.
%\end{array}
%\end{equation}

\subsection{Epitomic Matching for Internal Examples}

\subsubsection{The High Frequency Transfer Scheme}
Based on the observation that singular features like edges and corners in small patches tend to repeat almost identically across different image scales, Freedman and Fattal \cite{Fattal2010} applied the ``high frequency transfer'' method to searching the high-frequency component for a target HR patch, by NN patch matching across scales. Defining a linear interpolation operator $\mathcal{U}$ and a downsampling operator $\mathcal{D}$, for the input LR image $\mathbf{Y}$, we first obtain its initial upsampled image $\mathbf X^{'E} = \mathcal{U} (\mathbf{Y})$, and a smoothed input image $\mathbf{Y'} = \mathcal{D} (\mathcal{U} (\mathbf{Y}))$. Given the smoothed patch $\mathbf X_{ij}^{'E}$, the missing high-frequency band of each unknown patch $\mathbf X_{ij}^{E}$ is predicted by first solving a NN matching (\ref{freedman}):
\begin{equation}
\begin{array}{l}\label{freedman}
(m, n) = \arg\min_{(m, n) \in \mathcal{W}_{ij}} \| \mathbf Y'_{mn} - \mathbf X_{ij}^{'E}\|_F^2,
\end{array}
\end{equation}
where $\mathcal{W}_{ij}$ is defined as a small local searching window on image $\mathbf{Y'}$. We could also simply express it as $(m, n) = f_{NN} (\mathbf X_{ij}^{'E}, \mathbf{Y}')$. With the co-located patch $\mathbf{Y}_{mn}$ from $\mathbf{Y}$, the high-frequency band $\mathbf{Y}_{mn} - \mathbf{Y'}_{mn}$ is pasted onto $\mathbf X_{ij}^{'E}$, i.e., $\mathbf X_{ij}^{E} = \mathbf X_{ij}^{'E} + \mathbf{Y}_{mn} - \mathbf{Y'}_{mn}$.

% (with scaling factor $s$)

\subsubsection{EPI: Epitomic Matching for Internal SR}

The matching of $\mathbf X_{ij}^{'E}$ over the smoothed input image $\mathbf{Y'}$ makes the core step of the high frequency transfer scheme. However, the performance of NN matching (\ref{freedman}) is degraded with the presence of noise and outliers. Moreover, the NN matching in \cite{Fattal2010} is restricted to a local window for efficiency, which potentially accounts for some rigid artifacts.

Instead, we propose \textit{epitomic matching} to replace NN matching in the above frequency transfer scheme. As a generative model, epitome \cite{ChuYLCH10, Yang14} summarizes a large set of raw image patches into a condensed representation in a way similar to Gaussian Mixture Models. We first learn an epitome $\ve_{\mathbf{Y'}}$ from $\mathbf{Y'}$, and then match each $\mathbf X_{ij}^{'E}$ over $\ve_{\mathbf{Y'}}$ rather than $\mathbf{Y'}$ directly. Assume $(m, n) = f_{ept} (\mathbf X_{ij}^{'E}, \ve_{\mathbf{Y'}})$, where $ f_{ept}$ denotes the procedure of epitomic matching by $\ve_{\mathbf{Y'}}$. It then follows the same way as in \cite{Fattal2010}: $\mathbf X_{ij}^{E} = \mathbf X_{ij}^{'E} + \mathbf{Y}_{mn} - \mathbf{Y'}_{mn}$: the only difference here is the replacement of $f_{NN}$ with $f_{ept}$. The high-frequency transfer scheme equipped with epitomic matching can thus be applied to SR by itself as well, named \textit{EPI} for short, which will be included in our experiments in Section 4 and compared to the method using NN matching in \cite{Fattal2010}. 

Since $\ve_{\mathbf{Y'}}$ summarizes the patches of the entire $\mathbf{Y'}$, the proposed epitomic matching benefits from non-local patch matching. In the absence of self-similar patches in the local neighborhood, epitomic matching weights refer to non-local matches, thereby effectively reducing the artifacts arising from local matching \cite{Fattal2010} in a restricted small neighborhood.  In addition, note that each epitome patch summarizes a batch of similar raw patches in $\mathbf{Y'}$. For any patch $\mathbf{Y'}_{ij}$ that contains certain noise or outliers in $\mathbf{Y'}$, its has a small posterior and thus tends not be selected as candidate matches for $\mathbf X_{ij}^{'E}$, improving the robustness of matching. The algorithm details of epitomic matching are included in Appendix.

Moreover, we can also incorporate Nearest Neighbor (NN) matching to our epitomic matching, leading to a enhanced patch matching scheme that features both non-local (by epitome) and local (by NN) matching. Suppose the high frequency components obtained by epitomic matching and NN matching for patch $\mathbf X_{ij}^{'E}$ are ${\mathbf H}_{ij,\ve}$ and ${\mathbf H}_{ij,{\rm NN}}$ respectively, we use a smart weighted average of the two as the final high frequency component ${\mathbf H}_{ij}$:
\begin{align}\label{eq:weighted-frequency}
&{\mathbf H}_{ij} = w{\mathbf H}_{ij,\ve}+(1-w){\mathbf H}_{ij,{\rm NN}}
\end{align}
where the weight $w=p( {\vT^*_{ij}|{\mathbf X_{ij}^{'E}},\ve })$ denotes the probability of the most probable hidden mapping given the patch $\mathbf X_{ij}^{'E}$. A higher $w$ indicates that the patch $\mathbf X_{ij}^{'E}$ is more likely to have a reliable match by epitomic matching (with the probability measured through the corresponding most probable hidden mapping), thereby a larger weight is associated with the epitomic matching, and vice versa. This is the practical implementation of EPI that we used in the paper.

%In Section 4, we will include the internal SR method using NN \cite{Fattal2010} and EPI in our experiments, and compare their performances. 

Finally, we let $\Theta_I = \mathbf{X}_{ij}^E$ and define
\begin{equation}
\begin{array}{l}\label{li}
\ell_\mathcal{I} (\mathbf{X}_{ij}, \mathbf{X}_{ij}^E) = || \mathbf{X}_{ij} - \mathbf{X}_{ij}^E||_F^2,
\end{array}
\end{equation}
where $\mathbf{X}_{ij}^E$ is the internal SR result by epitomic matching.

%Similarly with in NN matching, It is straightforward to find K best matches (with the top K largest posterior probabilities) from epitomic matching, and to reconstruct K $\mathbf X_{ij}^{E}$'s. That will be utilized to solve our optimization problem in Section 3.4.

%Also note that the epitome training procedure can be significantly accelerated by GPU parallel computing, since the computation on each patch is independent.


%The statistics in \cite{Harold} further proved that there is no trivial rule to decide whether to use external or internal denoising on a patch-by-patch basis, and a neural network was learned to combine their effects.

\subsection{Learning the Adaptive Weights}
In \cite{Irani},  Mosseri et.al. showed that the internal versus external preference is tightly related to the Signal-to-Noise-Ratio (SNR) estimate of each patch. Inspired by that finding, we could seek similar definitions of "noise" in SR based on the latent representation errors. The \textit{external noise} is defined by the residual of sparse coding
\begin{equation}
\begin{array}{l}\label{ng}
N_g (\mathbf{a}_{ij}) =  || \mathbf{D_l}  \mathbf{a} _{ij}-  \mathbf{Y}_{ij}||_F^2.\\
\end{array}
\end{equation}
Meanwhile, the \textit{internal noise} finds its counterpart definition by the epitomic matching error within $f_{pet}$:
\begin{equation}
\begin{array}{l}\label{ni}
N_i (\mathbf{X}_{ij}^E) = || \mathbf Y'_{mn} - \mathbf X_{ij}^{'E}||_F^2,
\end{array}
\end{equation}
\noindent where $\mathbf Y'_{mn}$ is the matching patch in $\mathbf Y'$ for $\mathbf X_{ij}^{'E}$.

Usually, the two ``noises'' are on the same magnitude level, which aligns with the fact that external- and internal-examples will have similar performances on many (such as homogenous regions). However, there do exist patches where the two have significant differences in performances, as shown in Fig. \ref{fig:1}, which means the patch has a strong preference toward one of them. In such cases, the ``preferred'' term needs to be sufficiently emphasized. We thus construct the following patch-wise adaptive weight ($p$ is the hyperparameter):
\begin{equation}
\begin{array}{l}\label{C}
\omega (\mathbf{\alpha}_{ij}, \mathbf{X}_{ij}^E) = \exp(p\cdot [N_g (\mathbf{a}_{ij}) - N_i (\mathbf{X}^E_{ij}) ]).
\end{array}
\end{equation}
When the internal noise becomes larger, the weight decays quickly to ensure that external similarity dominates, and vice versa. 

\subsection{Optimization}
Directly solving (\ref{jsr}) is very complex due to the its high nonlinearity and entanglement among all variables. Instead, we follow the coordinate descent fashion \cite{Bertsekas1999} and solve the following three sub-problems iteratively.

\subsubsection{$\mathbf{a}_{ij}$-subproblem}
Fixing $\mathbf{X}_{ij}$ and  $\mathbf{X}_{ij}^E$, we have the following minimization w.r.t $ \mathbf{\alpha}_{ij}$
\begin{equation}
\begin{array}{l}\label{lg1}
\underset{\mathbf{a}_{ij}}{\min} \quad  \lambda ||\mathbf{a}_{ij}||_1 +   ||\mathbf{D_l} \mathbf{a}_{ij} - \mathbf{Y}_{ij}||_F^2 + ||\mathbf{D_h} \mathbf{a}_{ij} - \mathbf{X}_{ij}||_F^2  \\
\quad+ [\ell_\mathcal{I}(\mathbf{X}_{ij}, \mathbf{X}_{ij}^E) \cdot \exp( - p \cdot N_i (\mathbf{X}_{ij}^E)) ] \cdot \exp(p\cdot N_g (\mathbf{a}_{ij}) ).
\end{array}
\end{equation}
The major bottleneck of  exactly solving (\ref{lg1}) lies in the last exponential term. We let $\mathbf{a}^0_{ij}$ denote the $\mathbf{a}_{ij}$ value solved in the last iteration. We then apply first-order Taylor expansion to the last term of the objective in (\ref{lg1}), with regard to $N_g (\mathbf{\alpha}_{ij})$ at $\alpha_{ij}=\alpha^0_{ij}$, and solve the approximated problem as follows:
\begin{equation}
\begin{array}{l}\label{lg2}
\underset{\mathbf{a}_{ij}}{\min}  \quad \lambda ||\mathbf{a}_{ij}||_1 +  (1+C) ||\mathbf{D_l} \mathbf{a}_{ij} - \mathbf{Y}_{ij}||_F^2 + ||\mathbf{D_h} \mathbf{a}_{ij} - \mathbf{X}_{ij}||_F^2,
\end{array}
\end{equation}
where $C$ is the constant coefficient:
\begin{equation}
\begin{array}{l}\label{C}
C = [\ell_\mathcal{I}(\mathbf{X}_{ij}, \mathbf{X}_{ij}^E) \cdot \exp( - p \cdot N_i (\mathbf{X}_{ij}^E) ] \cdot [p\cdot \exp(p\cdot N_g (\mathbf{a}_{ij}^0)]\\
\quad  = p \ell_\mathcal{I}(\mathbf{X}_{ij}, \mathbf{X}_{ij}^E) \cdot \omega (\mathbf{\alpha^0}_{ij}, \mathbf{X}_{ij}^E).
\end{array}
\end{equation}
(\ref{lg2}) can be conveniently solved by the feature sign algorithm \cite{Lee2007}. Note (\ref{lg2}) is a valid approximation of (\ref{lg1}) since $\mathbf{a}_{ij}$ and $\mathbf{a}^0_{ij}$ become quite close after a few iterations, so that the higher-order Taylor expansions can be reasonably ignored. 

Another noticeable fact is that since $C>0$, the second term is always emphasized more than the third term, which makes sense as $\mathbf{Y}_{ij}$ is the ``accurate'' LR image, while $\mathbf{X}_{ij}$ is just an estimate of the HR image and is thus less weighted. Further considering the formulation (\ref{C}), $C$ grows up as $\omega (\mathbf{\alpha^0}_{ij}, \mathbf{X}_{ij}^E)$ turns larger. That implies when external SR becomes the major source of ``SR noise'' on a patch in the last iteration,  (\ref{lg2}) will accordingly rely less on the last solved $\mathbf{X}_{ij}$.


\subsubsection{$\mathbf{X}_{ij}^E$-subproblem}
Fixing $\mathbf{a}_{ij}$ and  $\mathbf{X}_{ij}$, the $\mathbf{X}_{ij}^E$-subproblem becomes
\begin{equation}
\begin{array}{l}\label{xe}
\underset{\mathbf{X}^E_{ij}}{\min} \quad \exp( -p \cdot || \mathbf Y'_{mn} - \mathbf X_{ij}^{'E}||_F^2) \ell_\mathcal{I}(\mathbf{X}_{ij},  \mathbf{X}^E_{ij}),
\end{array}
\end{equation}
%Note that (\ref{xe}) is in essence a reweighed version of epitomic matching described in Section 3.2, and could be solved by similar algorithms. 

While in Section III.B.2, $X_{i,j}^E$ is directly computed from the input LR image, the objective in (\ref{xe}) is dependent on not only $X_{i,j}^E$ but also $X_{i,j}$, which is not necessarily minimized by the best match $X_{i,j}^E$ obtained from solving $f_{ept}$. In our implementation, the $K$ best candidates ($K$ = 5) that yield minimum matching errors of solving $f_{ept}$ are first obtained.  Among all those candidates, we further select the one that minimizes the loss value as defined in (\ref{xe}). By this discrete search-type algorithm, $X_{i,j}^E$ becomes a latent variable to be updated together with $X_{i,j}$ per iteration, and is better suited for the global optimization than the simplistic solution by solving $f_{ept}$. 

%Because any $\mathbf{X}_{ij}^E$ minimizing $\ell_\mathcal{I}(\mathbf{X}_{ij},  \mathbf{X}^E_{ij})$ does not necessarily minimize $W(\mathbf{a}_{ij}, \mathbf{X}_{ij}^E)$, solving (\ref{xe}) exactly is intractable and we turn to an approximation solution by discrete search. First, $K$ best match candidates (with $K$-largest posterior probabilities) are found by epitomic matching, as mentioned in Section 3.2. Then, we substitute each of the $K$ matches into the objective of  (\ref{xe}), and choose one that leads to the minimum objective value as $\mathbf{X}_{ij}^E$.


\subsubsection{$\mathbf{X}_{ij}$-subproblem}
With both $\mathbf{a}_{ij}$ and  $\mathbf{X}_{ij}^E$ fixed, the solution of $\mathbf{X}_{ij}$ simply follows a weight least square (WLS) problem:
\begin{equation}
\begin{array}{l}\label{wls}
\underset{\mathbf{X}_{ij}}{\min} \quad   ||\mathbf{D_h} \mathbf{a}_{ij} - \mathbf{X}_{ij}||_F^2  + \omega(\mathbf{a}_{ij}, \mathbf{X}_{ij}^E) || \mathbf{X} - \mathbf{X}_{ij}^E||_F^2,
\end{array}
\end{equation}
with an explicit solution:
\begin{equation}
\begin{array}{l}\label{wls2}
\mathbf{X}_{ij} = \frac{\mathbf{D_h} \mathbf{a}_{ij} + \omega(\mathbf{\alpha}_{ij}, \mathbf{X}_{ij}^E)\cdot  \mathbf{X}_{ij}^E}{1+ \omega(\mathbf{a}_{ij}, \mathbf{X}_{ij}^E)}.
\end{array}
\end{equation}



%\begin{figure*}[htbp]
%\centering
%\begin{minipage}{0.30\textwidth}
%\centering \subfigure[BCI (PSNR = 20.9525 dB)] {
%\includegraphics[width=\textwidth]{1_interp.jpg}
%}\\
%\subfigure[CSC (PSNR =  23.4748 dB)] {
%\includegraphics[width=\textwidth]{1_input_small_csr.jpg}
%}\end{minipage}
%\begin{minipage}{0.30\textwidth}
%\centering \subfigure[LSE (PSNR = 22.8908 dB)] {
%\includegraphics[width=\textwidth]{1_Fattal.jpg}
%}\\
%\subfigure[EPI (PSNR = 21.9016 dB)] {
%\includegraphics[width=\textwidth]{1_epitome.jpg}
%}\end{minipage}
%\begin{minipage}{0.30\textwidth}
%\centering \subfigure[IER (PSNR = 22.7443 dB)] {
%\includegraphics[width=\textwidth]{1_input_small_photoshop.jpg}
%}\\
%\subfigure[JSR (PSNR = 24.1734 dB)] {
%\includegraphics[width=\textwidth]{1_jsr.jpg}
%}\end{minipage}
%\caption{4$\times$ SR result of the \textit{Kid} image.}
%\label{fig:kid}
%\end{figure*}


\begin{figure*}[htbp]
\centering
\begin{minipage}{0.40\textwidth}
\centering \subfigure[BCI (PSNR = 25.29 dB, SSIM = 0.8762)] {
\includegraphics[width=\textwidth]{temple_bicubic.jpg}
}\\
\subfigure[CSC (PSNR = 26.20 dB, SSIM = 0.8924)] {
\includegraphics[width=\textwidth]{temple_csc.jpg}
}\\
\subfigure[LSE (PSNR = 21.17 dB, SSIM = 0.7954)] {
\includegraphics[width=\textwidth]{temple_fattal.jpg}
}\\
\subfigure[EPI (PSNR = 24.34 dB, SSIM = 0.8901)] {
\includegraphics[width=\textwidth]{temple_epi.jpg}
}\\
\end{minipage}
\begin{minipage}{0.40\textwidth}
\centering \subfigure[IER (PSNR = 25.54 dB, SSIM = 0.8937)] {
\includegraphics[width=\textwidth]{temple_ier.jpg}
}\\
\subfigure[JSR (PSNR = 27.87 dB, SSIM = 0.9327)] {
\includegraphics[width=\textwidth]{temple_jsr.jpg}
}\\
\subfigure[Groundtruth] {
\includegraphics[width=\textwidth]{Temple_GT.jpg}
}\\
\subfigure[LR input] {
\includegraphics[width=\textwidth]{Temple_LR.png}
}\end{minipage}
\caption{3$\times$ SR results of the \textit{Temple} image. }
\label{fig:temple}
\end{figure*}



\section{Experiments}

\subsection{Implementation Details}

We itemize the parameter and implementation settings for the following group of experiments:
\begin{itemize}
\item  We use $5 \times 5$ patches with one pixel overlapping for all experiments except those on SHD images in Section 4.4, where the patch size is  $25 \times 25$ with five pixel overlapping.
\item In (\ref{lg}), we adopt the $\mathbf{D_l}$ and $\mathbf{D_h}$ trained in the same way as in \cite{Yang2012}, due to the similar roles played by the dictionaries in their formulation and our $\ell_G$ function. However, we are aware that such $D_l$ and $D_h$ are not optimized for the proposed method, and will integrate a specifically designed dictionary learning part in future work. $\lambda$ is empirically set as 1.
\item In (\ref{li}), the size of the epitome is $\frac{1}{4}$ of the image size. 
\item In (\ref{C}), we set $p=1$ for all experiments. We also observed in experiments that a larger $p$ will usually lead to a faster decrease in objective value, but the SR result quality may degrade a bit.
\item  We initialize $\mathbf{a}_{ij}$ by solving coupled sparse coding in \cite{Yang2012}. $\mathbf{X}_{ij}$ is initialized by bicubic interpolation.
 \item We set the maximum iteration number to be 10 for the coordinate descent algorithm. For SHD cases, the maximum iteration number is adjusted to be 5.
 \item For color images, we apply SR algorithms to the illuminance channel only, as humans are more sensitive to illuminance changes. We then interpolate the color layers (Cb, Cr) using plain bi-cubic interpolation.
 \end{itemize}
 


%Further, we visualize the patch-wise weight maps of joint SR on the \textit{Kid} image, at iterations 1, 3, 5, 7, and 10, as heat maps in Fig. \ref{fig:weight}. The weights start with a relatively scattered distribution at the very beginning. Yet during iterations, most homogenous regions, as well as the eyes and nose, are associated with rather small weights, showing that the external similarity is able to gain more advantages there. The larger weight values become sparse and significantly focused on the repetitive textures, e.g. the sweater, where the self-similarity will be imposed more heavily. Note the observation is in accordance with our intuition in Section 2. We also note that the weight map changes little from iteration 7 to iteration 10, which implies that the joint SR algorithm reaches a stable solution after a few iterations.

%\begin{figure}[htbp]
%\centering
%\includegraphics[width=0.4 \textwidth]{Graph1.jpg}
%\caption{The PSNR curve of the \textit{Kid} image: (a) with different fixed global weights; (b) with an adaptive weight. }
%\label{fig:kidadaptive}
%\end{figure}



%\begin{figure*}[htbp]
%\centering
%\begin{minipage}{0.18\textwidth}
%\centering \subfigure[] {
%\includegraphics[width=\textwidth]{iter1.jpg}
%}\end{minipage}
%\begin{minipage}{0.18\textwidth}
%\centering \subfigure [] {
%\includegraphics[width=\textwidth]{iter3.jpg}
%}\end{minipage}
%\begin{minipage}{0.18\textwidth}
%\centering \subfigure [] {
%\includegraphics[width=\textwidth]{iter5.jpg}
%}\end{minipage}
%\begin{minipage}{0.18\textwidth}
%\centering \subfigure [] {
%\includegraphics[width=\textwidth]{iter7.jpg}
%}\end{minipage}
%\begin{minipage}{0.18\textwidth}
%\centering \subfigure [] {
%\includegraphics[width=\textwidth]{iter10.jpg}
%}\end{minipage}
%\caption{The weight map produced by joint SR on the \textit{Kid} image at (a) iteration 1; (b) iteration 3; (c) iteration 5; (d) iteration 7; and (e) iteration 10.}
%\label{fig:weight}
%\end{figure*}


\subsection{Comparison with State-of-the-Art Results}

We compare the proposed method with the following selection of competitive methods as follows,
\begin{itemize}
\item  \textit{Bi-Cubic Interpolation (``BCI'' for short and similarly hereinafter)}, as a comparison baseline.
\item \textit{Coupled Sparse Coding (CSC)} \cite{Yang2012}, as the classical external-example-based SR.
\item \textit{Local Self-Example based SR (LSE)} \cite{Fattal2010}, as the classical internal-example-based SR.
\item \textit{Epitome-based SR (EPI)}. We compare EPI to LSE to demonstrate the advantage of epitomic matching over the local NN matching.
\item \textit{SR based on In-place Example Regression (IER)} \cite{Yang2013}, as the previous SR utilizing both external and internal information.
\item \textit{The proposed joint SR (JSR)}.
\end{itemize}
We list the SR results (best viewed on a high-resolution display) for two test images: \textit{Temple} and \textit{Train}, by an amplifying factor of 3. PSNR and SSIM measurements, as well as zoomed local regions (using nearing neighbor interpolation), are available for different methods as well. 

\begin{figure*}[htbp]
\centering
\begin{minipage}{0.40\textwidth}
\centering \subfigure[BCI (PSNR = 26.14 dB, SSIM = 0.9403)] {
\includegraphics[width=\textwidth]{train_bicubic.jpg}
}\\
\subfigure[CSC (PSNR = 26.58 dB, SSIM = 0.9506)] {
\includegraphics[width=\textwidth]{train_csc.jpg}
}\\
\subfigure[LSE (PSNR = 22.54 dB, SSIM = 0.8850)] {
\includegraphics[width=\textwidth]{train_fattal.jpg}
}\\
\subfigure[EPI (PSNR = 26.22 dB, SSIM = 0.9487)] {
\includegraphics[width=\textwidth]{train_epitome.jpg}
}\end{minipage}
\begin{minipage}{0.40\textwidth}
\centering\subfigure[IER (PSNR = 24.80 dB, SSIM = 0.9323)] {
\includegraphics[width=\textwidth]{train_ier.jpg}
}\\
\subfigure[JSR (PSNR = 28.02 dB, SSIM = 0.9796)] {
\includegraphics[width=\textwidth]{train_jsr.jpg}
}\\
\subfigure[Groundtruth] {
\includegraphics[width=\textwidth]{Train_GT.jpg}
}\\
\subfigure[LR input] {
\includegraphics[width=\textwidth]{Train_LR.png}
}\end{minipage}
\caption{3$\times$ SR results of the \textit{Train} image. }
\label{fig:train}
\end{figure*}



%We generate SR results for CSC and IER using their original codes \cite{Yang2012}, \cite{Yang2013}, and implement LSE \cite{Fattal2010} to the best of our ability.

%In this section, we magnify all the input LR images by a factor of 4.  As the visual quality is the most important criterion for evaluating SR, we list the visual results (best viewed on a high-resolution display), from Fig. \ref{fig:kid} to Fig. \ref{fig:statue}. Detailed comparisons can be found at the enlarged local regions. Since we only have access to the ground truth HR image of the \textit{Kid} image, we list the PSNR measurements in Figure \ref{fig:kid} as well.

In Fig. \ref{fig:temple}, although greatly outperforming the naive BCI, the external-example based CSC tends to lose many fine details. In contrast, LSE brings out an overly sharp SR result with observable blockiness. EPI produces a more visually pleasing result, through searching for the matches over the entire input efficiently by the pre-trained epitome rather than a local neighborhood. Therefore, EPI substantially reduces the artifacts compared to LSE. But without any external information available, it is still incapable of inferring enough high-frequency details from the input solely, especially under a large amplifying factor. The result of IER greatly improves but is still accompanied with occasional small artifacts. Finally, JSR provides a clear recovery of the steps, and it reconstructs the most pillar textures. In Fig. \ref{fig:train},  JSR is the only algorithm which clearly recovers the number on the carrier and the bricks on the bridge simultaneously. The performance superiorities of JSR are also verified by the PSNR comparisons, where larger margins are obtained by JSR over others in both cases.

%JSR is the only algorithm that can reconstruct the pillar textures. Figure \ref{fig:kid} shows the SR results for the \textit{Kid} image. Although outperforming the naive BCI noticeably, the external-example based CSC tends to oversmoothen many fine details in the sweater, the eyelash, and so on. In contrast, LSE brings out an overly sharp SR result, suffering from observable blockiness in hat regions, inconsistency in eyelids, as well as ringings and jaggies along the edges of faces and eyeballs. Compared the LSE, EPI produces a more visually pleasing result with a part of the artifacts removed. But without any external information available, EPI is still incapable of inferring enough high-frequency details from the input LR image solely, especially under a large amplifying factor. The result of IER greatly improves but is still accompanied with occasional small artifacts. One may even argue that its eye and hat parts are not as naturally-looking as in CSC. Finally, JSR leads to the most visually satisfactory result. It emphasizes external examples for regular regions (face, eyeballs, etc.), avoiding rigid artifacts caused by the bad matches of internal examples. For singular features (eyelash, etc.), JSR adaptively shows a more similar performance to LSE/EPI and gives out clear details resulting from internal examples. Its performance superiority is also verified by the PSNR comparison. 

Next, we move on to the more challenging 4$\times$ SR case, using the \textit{Chip} image which is quite abundant in edges and textures. Since we have no ground truth for the \textit{Chip} image of 4$\times$ size, only visual comparisons are presented. Given such a large SR factor, the CSC result is a bit blurry around the characters on the surface of chip. Both LSE and EPI create jaggy artifacts along the long edge of the chip, as well as small structure distortions. The IER result cause less artifacts but in sacrifice of detail sharpness. The JSR result presents the best SR with few artifacts. 

The key idea of JSR is utilizing the complementary behavior of both external and internal SR methods. Note when one inverse problem is better solved, it also makes a better parameter estimate for solving the other. JSR is not a simple static weighted average of external SR (CSC) and internal SR (EPI). When optimized jointly, the external and internal subproblems can "boost" each other (through auxiliary variables), and each performs better than being applied independently. That is why JSR gets details that exist in neither internal or external SR result.

To further verify the superiority of JSR numerically, we compare the average PSNR and SSIM results of a few recently-proposed, state-of-the-art single image SR methods, including CSC, LSE, the Adjusted Anchored Neighborhood Regression (A+) \cite{A+}, and the latest Super-Resolution Convolutional Neural Network (SRCNN) \cite{Tang}. Table \ref{set} reports the results on the widely-adopted Set 5 and Set 14 datasets, in terms of both PSNR and SSIM. First, it is not a surprise to us, that JSR does not always yield higher PSNR than SRCNN, et. al., as the epitomic matching component is not meant to be optimized under Mean-Square-Error (MSE) measure, in contrast to the end-to-end MSE-driven regression adopted in SRCNN. However, it is notable that JSR is particularly more favorable by SSIM than other methods, owing to the self-similar examples that convey input-specific structural details. Considering SSIM measures image quality more consistently with human perception, the observation is in accordance with our human subject evaluation results (see Section IV. E). 


\begin{table*}[t]
\begin{center}
\caption{Average PSNR (dB) and SSIM performances comparisons on the Set 5 and Set 14 datasets}
\label{set}
\vspace{0.5em}
\begin{tabular}{|c|c|c|c|c|c|c|c|}
\hline
 & & Bicubic & Sparse Coding \cite{Yang2010} & Freedman et.al. \cite{Fattal2010}& A+ \cite{A+} & SRCNN \cite{Tang} &  JSR \\
% & & & + SC & & + SC& & + SC& UM & max-margin\\
\hline
$\multirow{2}{*}{\textit{Set 5, $s_t$=2}}$ &PSNR  & 33.66 & 35.27 & 33.61 & 36.24 & 36.66 & \textbf{36.71}\\
\cline{2-8}
$$ &SSIM & 0.9299 & 0.9540 & 0.9375 & 0.9544 & 0.9542 & \textbf{0.9573} \\
\hline
$\multirow{2}{*}{\textit{Set 5, $s_t$=3}}$ &PSNR & 30.39 & 31.42 & 30.77 & 32.59 & \textbf{32.75} & 32.54\\
\cline{2-8}
$$ &SSIM  & 0.8682 & 0.8821 & 0.8774 &0.9088 & 0.9090 & \textbf{0.9186} \\
\hline
$\multirow{2}{*}{\textit{Set 14, $s_t$=2}}$ &PSNR & 30.23 & 31.34 & 31.99 & \textbf{32.58} & 32.45& 32.54  \\
\cline{2-8}
$$ &SSIM & 0.8687 & 0.8928 & 0.8921 & 0.9056 & 0.9067 & \textbf{0.9082}  \\
\hline
$\multirow{2}{*}{\textit{Set 14, $s_t$=3}}$ &PSNR & 27.54 & 28.31 & 28.26 & 29.13 &\textbf{29.60} & 29.49  \\
\cline{2-8}
$$ &SSIM  & 0.7736 & 0.7954 & 0.8043 & 0.8188 & 0.8215 & \textbf{0.8242} \\
\hline
\end{tabular}
\end{center}
\end{table*}





\begin{figure*}[htbp]
\centering
\begin{minipage}{0.40\textwidth}
\centering \subfigure[BCI] {
\includegraphics[width=\textwidth]{5_interp.jpg}
}\\
\subfigure[CSC] {
\includegraphics[width=\textwidth]{5_input_small_csr.jpg}
}\\
\subfigure[LSE] {
\includegraphics[width=\textwidth]{5_Fattal.jpg}
}\end{minipage}
\begin{minipage}{0.40\textwidth}
\centering \subfigure[EPI] {
\includegraphics[width=\textwidth]{5_epitome.jpg}
}\\
\subfigure[IER] {
\includegraphics[width=\textwidth]{5_input_small_photoshop.jpg}
}\\
\subfigure[JSR] {
\includegraphics[width=\textwidth]{5_jsr.jpg}
}\end{minipage}
\centering \subfigure[LR input] {
\includegraphics[width=0.10\textwidth]{Chip_LR.png}
}
\caption{4$\times$ SR results of the \textit{Chip} image.}
\label{fig:chip}
\end{figure*}



\begin{figure*}[htbp]
\centering
\begin{minipage}{0.32\textwidth}
\centering \subfigure[Temple] {
\includegraphics[width=\textwidth]{temple.jpg}
}\end{minipage}
\begin{minipage}{0.32\textwidth}
\centering\subfigure[Train] {
\includegraphics[width=\textwidth]{train.jpg}
}\end{minipage}
\begin{minipage}{0.32\textwidth}
\centering \subfigure[Chip] {
\includegraphics[width=\textwidth]{chip.jpg}
}\end{minipage}
\caption{The weight maps of (a) \textit{Temple} image; (b) \textit{Train} image; (c) \textit{Chip} image.}
\label{fig:map}
\end{figure*}


%Similar comparisons could be found in Fig. \ref{fig:statue} of the \textit{Statue} image, where the differences in SR performance can be easily identified in the enlarged textured areas.

%
%the global complementary behavior
%
%Note we could equivalently express the weight in the form of sigmoid function:
%\begin{equation}
%\begin{array}{l}\label{C}
%\omega_{sig} (\mathbf{\alpha}_{ij}, \mathbf{X}_{ij}^E) = \frac{1}{1+\exp(p\cdot [N_g (\mathbf{a}_{ij}) - N_i (\mathbf{X}^E_{ij}) ])},
%\end{array}
%\end{equation}
%so that the objective (\ref{jsr}) can be alternatively written as a symmetric form:
%\begin{equation}
%\begin{array}{l}\label{jsr2}
%\underset{\mathbf{X}_{ij}, \mathbf{\alpha}_{ij}, \mathbf{X}_{ij}^E}{\min}  \omega_{sig} \ell_\mathcal{G} (\mathbf{X}_{ij}, \mathbf{a}_{ij}) + (1-\omega_{sig}) \ell_\mathcal{I}(\mathbf{X}_{ij},  \mathbf{X}_{ij}^E).
%\end{array}
%\end{equation}

 \subsection{Effect of Adaptive Weight}

To demonstrate how the proposed joint SR will benefit from the learned adaptive weight (\ref{C}), we compare 4$\times$ SR results of \textit{Kid} image, between joint SR solving (\ref{jsr}), and its counterpart with fixed global weights , i.e. set the weight $\omega$ as constant for all patches. Table 1 shows that the joint SR with an adaptive weight gains a consistent PSNR advantage over the SR with a large range of fixed weights. 

\begin{table}[t]
\begin{center}
\caption{The PSNR values (dB) with various fixed global weights (PSNR = 24.1734 dB with an adaptive weight)}
\vspace{0.1em}
\begin{tabular}{|c|c|c|c|c|}
\hline
$\omega$ = 0.1 & $\omega$ = 1 & $\omega$ = 3 & $\omega$ = 5 & $\omega$ = 10  \\
\hline
23.13  & 23.23  & 23.32  & 22.66 & 21.22 \\
\hline
\end{tabular}
\end{center}
\end{table}


More interestingly, we visualize the patch-wise weight maps of joint SR results in Fig. \ref{fig:temple} - \ref{fig:chip}, using heat maps, as in Fig. \ref{fig:map}. The ($i,j$)-th pixel in the weight map denote the final weight of $\mathbf{X}_{ij}$ when the joint SR reaches a stable solution. All weights are normalized between [0,1], by the form of sigmoid function: $\frac{1}{1+ \omega(\mathbf{\alpha}_{ij}, \mathbf{X}_{ij}^E)}$, for visualization purpose. A larger pixel value in the weight maps denote a smaller weight and thus a higher emphasis on external examples, and vice versa. For \textit{Temple} image, Fig. \ref{fig:map} (a) clearly manifests that self examples dominate the SR of the temple building that is full of texture patterns. Most regions of Fig. \ref{fig:map} (b) are close to 0.5, which means that  $\omega (\mathbf{\alpha}_{ij}, \mathbf{X}_{ij}^E)$ is close to 1 and external and internal examples have similar performances on most patches. However, internal similarity makes more significant contributions in reconstructing the brick regions, while external examples works remarkably better on the irregular contours of forests. Finally, the \textit{Chip} image is an example where external examples have advantages on the majority of patches. Considering self examples prove to create artifacts here (see Fig. \ref{fig:chip} (c) (d)), they are avoided in joint SR by the adaptive weights.


%We further observe in experiments that with the optimization converging to a stable solution, the distribution of adaptive weights on all patches reaches the equilibrium too. The larger weight values will become sparse and significantly focused on the repetitive textures, which is in accordance with our intuition in Section 2.





%In total, the textures on the child's hat and sweaters are nicely preserved, while the fine structures of the eye parts are better reconstructed than others.


%We also make assessments for the four images by peak signal-to-noise ratio (PSNR). The PSNR values for all test images by different methods are listed in Table 1, and our algorithm outperforms all others in terms of PSNR.

\subsection{SR Beyond Standard Definition: From HD Image to UHD Image}

In almost all SR literature, experiments are conducted with Standard-Definition (SD) images (720 $\times$ 480 or 720 $\times$ 576 pixels) or smaller. The High-Definition (HD) formats:  720p (1280 $\times$ 720 pixels) and 1080p (1920 $\times$ 1080 pixels) have become popular today. Moreover, Ultra High-Definition (UHD) TVs are hitting the consumer markets right now with the 3840 $\times$ 2160 resolution. It is thus quite interesting to explore whether SR algorithms established on SD images can be applied or adjusted for HD or UHD cases. In this section, we upscale HD images of 1280 $\times$ 720 pixels to UHD results of 3840 $\times$ 2160 pixels, using competitor methods and our joint SR algorithm. 

%Today, the TV industry supports two popular High-Definition (HD) formats; 720p (1280 $\times$ 720 pixels) and 1080p (1920 $\times$ 1080 pixels, Full HD). All HDTVs come with HD resolution. Moreover, the TV industry is already pushing for Ultra High-Definition (UHD) standard, which covers both 4K/2160p (3840 $\times$ 2160 pixels) and 8K/4320p (7680 $\times$ 4320 pixels). 

\begin{figure*}[htbp]
\centering
\begin{minipage}{0.90\textwidth}
\centering \subfigure[Full SHD image] {
\includegraphics[width=\textwidth]{Lion.jpg}
}\end{minipage}
\begin{minipage}{0.45\textwidth}
\centering \subfigure[Local region from SR result by BCI \newline PSNR = 24.14 dB, SSIM = 0.9701] {
\includegraphics[width=\textwidth]{LionGT.jpg}
}
\subfigure [Local region from SR result by CSC \newline PSNR = 25.32 dB, SSIM = 0.9618] {
\includegraphics[width=\textwidth]{LionCSC.jpg}
}\end{minipage}
\begin{minipage}{0.45\textwidth}
\centering \subfigure [Local region from SR result by EPI \newline PSNR = 23.58 dB, SSIM = 0.9656] {
\includegraphics[width=\textwidth]{LionEPI.jpg}
}
\subfigure [Local region from SR result by JSR \newline PSNR = 25.82 dB, SSIM = 0.9746] {
\includegraphics[width=\textwidth]{LionJSR.jpg}
}\end{minipage}
\caption{3$\times$ SR results of the \textit{Leopard} image (local region displayed).}
\label{fig:Lion}
\end{figure*}

%\begin{figure*}[htbp]
%\centering
%\begin{minipage}{0.48\textwidth}
%\centering \subfigure[Local region from the original  image] {
%\includegraphics[width=\textwidth]{grass_gt_cropped.jpg}
%}
%\subfigure [Local region from SR result by CSC] {
%\includegraphics[width=\textwidth]{grass_csc_cropped.jpg}
%}\end{minipage}
%\begin{minipage}{0.48\textwidth}
%\centering \subfigure [Local region from SR result by EPI] {
%\includegraphics[width=\textwidth]{grass_epi_cropped.jpg}
%}
%\subfigure [Local region from SR result by JSR] {
%\includegraphics[width=\textwidth]{grass_jsr_cropped.jpg}
%}\end{minipage}
%\caption{3$\times$ SR results of the \textit{Grass} image (local region displayed).}
%\label{fig:Grass}
%\end{figure*}

%The Original SHD Image could be found in supplementary material.

Since most HD and UHD images typically contain much more diverse textures and a richer collection of fine structures than SD images, we enlarge the patch size from $5 \times 5$ to $25 \times 25$ (the dictionary pair is therefore re-trained as well) to capture more variations, meanwhile increasing the overlapping from one pixel to five pixels to ensure enough spatial consistency. Hereby JSR is compared with its two ``component'' algorithms, i.e., CSC and EPI.  We choose several challenging SHD images (3840 $\times$ 2160 pixels) with very cluttered texture regions, downsampling them to HD size (1280 $\times$ 720 pixel) on which we apply the SR algorithm with a factor of 3. In all cases, our results are consistently sharper and clearer. The SR results (zoomed local regions) of the \textit{Leopard} image are displayed in Fig. \ref{fig:Lion} for examples, with the PSNR and SSIM measurements of full-size results.

%The original SHD images and their SR results could be found in our supplementary material. 



%In both cases, our results are consistently sharper and clearer.


%SR Results on two HD images,  \textit{Leopard} and  \textit{Grass}, are displayed in Fig. \ref{fig:Lion} and Fig. \ref{fig:Grass} as zoomed local regions, while original results are included in supplementary materials (better viewed on UHD display). In both cases, our results are consistently sharper and clearer.





%grass mask: (500:1000,600:1200,:)

%The large body of natural images typically contain more diverse textures and richer fine structures. Fig.~\ref{fig:temple} and \ref{fig:train} show $3\times$ SR results on real-world images with cluttered texture regions. We again compare the proposed JSR with CSC and LSE. Again, CSC can produce natural looking but blurry results, while LSE fails to recover the fine details and produces many rigid false textures, both of which confirm again the different but complementary behaviors of external and internal examples. In both challenging cases, Our results are consistently sharper and clearer than CSC, and are free of the artifacts that LSE suffers. In Fig.~\ref{fig:temple}, JSR is the only algorithm that can reconstruct the pillar textures well. In Fig.~\ref{fig:train}, the pier region of JSR result contains the most faithful structural details.


\subsection{Subjective Evaluation}
We conduct an online \textit{subjective evaluation} survey \footnote {\url{http://www.ifp.illinois.edu/~wang308/survey}} on the quality of SR results produced by all different methods in Section 4.2. Ground truth HR images are also included when they are available as references. Each participant of the survey is shown a set of HR image pairs obtained using two different methods for the same LR image. For each pair, the participant needs to decide which one is better than the other in terms of perceptual quality. The image pairs are drawn from all the competitive methods randomly, and the images winning the pairwise comparison will be compared again in the next round, until the best one is selected. We have a total of 101 participants giving 1,047 pairwise comparisons, over six images which are commonly used as benchmark images in SR, with different scaling factors (\textit{Kid}$\times4$, \textit{Chip}$\times4$, \textit{Statue}$\times4$, \textit{Leopard}$\times3$, \textit{Temple}$\times3$ and \textit{Train}$\times3$). We fit a Bradley-Terry \cite{Bradley52rank} model to estimate the subjective scores for each method so that they can be ranked. More experiment details are included in our Appendix. Figure \ref{fig:subeval} shows the estimated scores for the six SR methods in our evaluation. As expected, all SR methods receive much lower scores compared to ground truth (set as score 1), showing the huge challenge of the SR problem itself. Also, the bicubic interpolation is significantly worse than others. The proposed JSR method outperforms all other state-of-the-art methods by a large margin, which proves that JSR can produce more visually favorable HR images by human perception.

%Not every participant completed all the comparisons but their partial responses are still useful.
%All the evaluation results can be summarized in a $7{\times}7$ winning matrix $\mathbf{W}$ for seven methods (including ground truth),
%based on which we fit a Bradley-Terry \cite{Bradley52rank} model to estimate
%the subjective scores for each method so that they can be ranked.
%In the Bradley-Terry model, the probability that an object $X$ is favored over $Y$ is assumed to be
%\begin{equation}
%\label{eq:bt}
%    p(X \succ Y) = \frac{e^{s_X}}{e^{s_X}+e^{s_Y}} = \frac{1}{1+e^{s_Y-s_X}} ,
%\end{equation}
%where $s_X$ and $s_Y$ are the subjective scores for $X$ and $Y$.
%The scores $\mathbf{s}$ for all the objects can be jointly estimated
%by maximizing the log likelihood of the pairwise comparison observations:
%\begin{equation}
%\label{eq:bt_mle}
%    \max_{\mathbf{s}} \sum_{i, j} w_{ij}\log \left(\frac{1}{1+e^{s_j-s_i}}\right) ,
%\end{equation}
%where $w_{ij}$ is the $(i, j)$-th element in the winning matrix $\mathbf{W}$,
%representing the number of times when method $i$ is favored over method $j$.
%We use the Newton-Raphson method to solve \eqref{eq:bt_mle} and set the score for
%ground truth as 1 to avoid the scale issue.

\begin{figure}[t]
\center
    \includegraphics[angle=0,width=70mm,clip]{score_jsr.jpg}
\caption{Subjective SR quality scores for different methods. The ground truth has score 1.}
\vspace{-3mm}
\label{fig:subeval}
\end{figure}





%\begin{table*}[t]
%\begin{center}
%\caption{PSNR and SSIM performance comparisons}
%\vspace{0.5em}
%\begin{tabular}{|c|c|c|c|c|c|c|c|}
%\hline
% & & BCI & CSC & LSE  & IER  &EPI & JSR \\
%\hline
%$\multirow{2}{*}{\textit{Kid}}$ &PSNR  & 29.78 & 33.83 & 29.87 & 32.06 & 34.24 &\textbf{35.88} \\
%\cline{2-8}
%$$ &SSIM  & 0.9045 & 0.9352 & 0.8832 & 0.9235 & 0.9550 &\textbf{0.9772} \\
%\hline
%$\multirow{2}{*}{\textit{?Chip}}$ &PSNR & 26.54 & 28.51 & 28.77 & 26.93 & 28.93 &\textbf{30.81}  \\
%\cline{2-8}
%$$ &SSIM  & 0.8554 & 0.8951 & 0.8774 &0.8871 &0. 8943 &\textbf{0.9181}  \\
%\hline
%$\multirow{2}{*}{\textit{?Statue}}$ &PSNR & 30.34 & 32.55 & 30.99 & 30.33 & 32.75 & \textbf{34.93}   \\
%\cline{2-8}
%$$ &SSIM  & 0.8554 & 0.8951 & 0.8774 &0.8871 &0. 8943 &\textbf{0.9181}  \\
%\hline
%$\multirow{2}{*}{\textit{Lion (SHD)}}$ &PSNR &- & 25.32 & - & -  & 23.58 &\textbf{25.82}  \\
%\cline{2-8}
%$$ &SSIM  & - & 0.8082 & -  & - & 0.8298  & \textbf{0.8684}  \\
%\hline
%\end{tabular}
%\end{center}
%\end{table*}

\section{Conclusion}
This paper presents a joint single image SR model, by learning from both external and internal examples. We define the two loss functions by sparse coding and epitomic matching, respectively, and construct an adaptive weight to balance the two terms. Experimental results demonstrate that joint SR outperforms existing state-of-the-art methods for various test images of different definitions and scaling factors, and is also significantly more favored by user perception. We will further integrate dictionary learning into the proposed scheme, as well as reducing its complexity.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\appendix
 
\textbf{1. Epitomic Matching Algorithm}
 
 We assume an epitome $\ve$ of size ${M_e} \times {N_e}$, for an input image of size $M \times N$, where ${M_e} < M$ and ${N_e} < N$. Similarly to GMMs, $\ve$ contains three parameters \cite{JojicFK03,Ni09,ChuYLCH10}: $\semean$, the Gaussian mean of size ${M_e} \times {N_e}$; $\sevar$, the Gaussian variance of size ${M_e} \times {N_e}$; and $\vpi$,  the mixture coefficients. Suppose $Q$ densely sampled, overlapped patches from the input image, i.e. $\{\vZ_k\}^Q_{k=1}$. Each $\vZ_k$ contains pixels with image coordinates $\vS_k$, and is associated with a hidden mapping $\vT_k$ from $\vS_k$ to the epitome coordinates. All the $Q$ patches are generated independently from the epitome and the corresponding hidden mappings as below:

\begin{align} \label{eq:patchfromT1}
&\prod_{k=1}^Q p(\{\vZ_k\}^Q_{k=1} | \{\vT_k\}^Q_{k=1}, \ve) = \prod_{k=1}^Q p(\vZ_k | \vT_k, \ve)
\end{align}

The probability $p(\vZ_k | \vT_k, \ve)$ in (\ref{eq:patchfromT1}) is computed by the Gaussian distribution where the Gaussian component is specified by the hidden mapping $\vT_k$. $\vT_k$ behaves similar to the hidden variable in the traditional GMMs.

\begin{figure}[htbp]
\begin{center}
\includegraphics[scale=.3]{epitome_patch.jpg}
\end{center}
   \caption{(a) The hidden mapping $\vT_k$ maps the image patch $\vZ_k$ to its corresponding patch of the same size in $\ve$, and $\vZ_k$ can be mapped to any possible epitome patch in accordance with $\vT_k$. (b) The epitome graphical model}
\label{fig:hiddenmapping}
\end{figure}

Figure~\ref{fig:hiddenmapping} illustrates the role that the hidden mapping plays in the epitome as well as the graphical model illustration for epitome. With all the above notations, our goal is to find the epitome $\ve$ that maximizes the log likelihood function ${{\ve}} = \mathop {\arg \max }\limits_{\hat\ve} \log p\left( {\{\vZ_k\}^Q_{k=1}|\hat\ve} \right)$, which can be solved by the Expectation-Maximization (EM) algorithm \cite{JojicFK03,Yang14}. 

The Expectation step in the EM algorithm which computes the posterior of all the hidden mappings accounts for the most time consuming part of the learning process. Since the posterior of the hidden mappings for all the patches are independent of each other, they can be computed in parallel. Therefore, the learning process can be significantly accelerated by parallel computing.

With the epitome $\ve_{\mathbf{Y'}}$ learned from the smoothed input image $\mathbf{Y'}$, the location of the matching patch in the epitome $\ve_{\mathbf{Y'}}$ for each patch $\mathbf X_{ij}^{'E}$  is specified by the most probable hidden mapping for $\mathbf X_{ij}^{'E}$:
\begin{align} \label{eq:mostprobT}
&\vT^*_{ij} = \mathop {\arg \max }\limits_{\vT_{ij}} p\left( {\vT_{ij}|{\mathbf X_{ij}^{'E}},\ve } \right)
\end{align}
The top $K$ patches in $\mathbf{Y'}$ with large posterior probabilities $p\left( {\vT^*_{ij}|{\cdot},\ve} \right)$ are regarded as the candidate matches for the patch $\mathbf{X'}_{ij}$, and the match $\mathbf{Y}_{mn}'$ is the one in these $K$ candidate patches which has minimum Sum of Squared Distance (SSD) to $\mathbf X_{ij}^{'E}$. Note that the indices of the $K$ candidate patches in $\mathbf{Y'}$ for each epitome patch are pre-computed and stored when training the epitome $\ve_{\mathbf{Y'}}$ from the smoothed input image $\mathbf{Y'}$, which makes epitomic matching efficient.

%Moreover, we can incorporate Nearest Neighbor (NN) matching to our epitomic matching, leading to a enhanced patch matching scheme that features both non-local (by epitome) and local (by NN) matching. Suppose the high frequency components obtained by epitomic matching and NN matching for patch $\mathbf X_{ij}^{'E}$ are ${\mathbf H}_{ij,\ve}$ and ${\mathbf H}_{ij,{\rm NN}}$ respectively, we use a smart weighted average of the two as the final high frequency component ${\mathbf H}_{ij}$:
%\begin{align}\label{eq:weighted-frequency}
%&{\mathbf H}_{ij} = w{\mathbf H}_{ij,\ve}+(1-w){\mathbf H}_{ij,{\rm NN}}
%\end{align}
%where the weight $w=p( {\vT^*_{ij}|{\mathbf X_{ij}^{'E}},\ve })$ which is probability of the most probable hidden mapping given the patch $\mathbf X_{ij}^{'E}$. A higher $w$ indicates that the patch $\mathbf X_{ij}^{'E}$ is more likely to have a match by epitomic matching with the probability measured through the corresponding most probable hidden mapping, thereby more weight is associated with the epitomic matching, and vice versa. This is the practical implementation of \textit{epitome-based SR} (EPI) we used in the experiment section of the submitted paper.

EPI significantly reduces the artifacts and produces more visually pleasing SR results by the dynamic weighting (\ref{eq:weighted-frequency}), compared to the local NN matching method {\cite{Fattal2010}}.

\textbf{2. Subjective Review Experiment}

The methods under comparison include BIC, CSC, LSE, IER, EPI, JSR.
Ground truth HR images are also included when they are available as references.
Each of the human subject participating in the evaluation is shown a set of HR image pairs obtained
using two different methods for the same LR image.
For each pair, the subject needs to decide which one is better than the other in terms of perceptual quality.
The image pairs are drawn from all the competitive methods randomly,
and the images winning the pairwise comparison will be compared again in the next round until the best one is selected.

We have a total of 101 participants giving 1,047 pairwise comparisons over 6 images with different scaling factors
(``Kid''$\times4$, ``Chip''$\times4$, ``Statue''$\times4$, ``Leopard''$\times3$, ``Temple''$\times3$ and ``Train''$\times3$).
Not every participant completed all the comparisons but their partial responses are still useful.
All the evaluation results can be summarized into a $7{\times}7$ winning matrix $\mathbf{W}$ for 7 methods (including ground truth),
based on which we fit a Bradley-Terry \cite{Bradley52rank} model to estimate
the subjective score for each method so that they can be ranked.
In the Bradley-Terry model, the probability that an object $X$ is favored over $Y$ is assumed to be
\begin{equation}
\label{eq:bt}
    p(X \succ Y) = \frac{e^{s_X}}{e^{s_X}+e^{s_Y}} = \frac{1}{1+e^{s_Y-s_X}} ,
\end{equation}
where $s_X$ and $s_Y$ are the subjective scores for $X$ and $Y$.
The scores $\mathbf{s}$ for all the objects can be jointly estimated
by maximizing the log likelihood of the pairwise comparison observations:
\begin{equation}
\label{eq:bt_mle}
    \max_{\mathbf{s}} \sum_{i, j} w_{ij}\log \left(\frac{1}{1+e^{s_j-s_i}}\right) ,
\end{equation}
where $w_{ij}$ is the $(i, j)$-th element in the winning matrix $\mathbf{W}$,
representing the number of times when method $i$ is favored over method $j$.
We use the Newton-Raphson method to solve Eq.~\eqref{eq:bt_mle} and set the score for
ground truth method as 1 to avoid the scale issue.

Fig.~\ref{fig:subeval} shows the estimated scores for six SR methods in our evaluation.
As expected, all the SR methods have much lower scores than ground truth, showing the great challenge in SR problem.
Also, the bicubic interpolation is significantly worse than other SR methods.
The proposed JSR method outperforms other previous state-of-the-art methods by a large margin,
which verifies that JSR can produce visually more pleasant HR images than other approaches.



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% BIBLIOGRAPHY
%
\begin{thebibliography}{1}
%\bibliographystyle{IEEE_ECE}

% Put references in BibTeX format in thesisrefs.bib.
\bibitem{Park2003} S. C. Park, M. K. Park, and K. G. Kang, ``Super-resolution image reconstruction: A technical overview,'' \textit{IEEE Signal Processing Magazine}, vol. 20, no. 3, pp. 21-36, 2003.

%\bibitem{Morse2003} B. Morse, D. Schwartzwald, ``Image Magnification Using Level Set Reconstruction,'' In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 333-341, (2001)

\bibitem{Fattal2007} R. Fattal, ``Image upsampling via imposed edge statistics,'' \textit{ACM Transactions on Graphics}, vol. 26, no. 3, pp. 95-102, 2007.

%\bibitem{Irani1991} M. Irani, S. Peleg, ``Improving Resolution by Image Registration,'' Computer Vision, Graphics and Image Processing, vol. 53, no. 3,  pp. 231-239, 1991

\bibitem{Lin2004} Z. Lin, and H. Y. Shum, ``Fundamental limits of reconstruction-based superresolution algorithms under local translation,'' \textit{IEEE Transactions on Pattern Analysis and Machine Intelligence}, vol. 26, no. 1, pp. 83-97, 2004.

\bibitem{Yang2012} J. Yang, Z. Wang, Z. Lin, S. Chen, and T. Huang, ``Coupled Dictionary Training for Image Super-resolution,'' \textit{IEEE Transactions on Image Processing}, vol. 21, no. 8, pp. 3467-3478, 2012.

\bibitem{Fattal2010} G. Freedman and R. Fattal, ``Image and video upscaling from local self-examples,''  \textit{ACM Transactions on Graphics}, vol. 28, no. 3, 2010.

\bibitem{JojicFK03} N. Jojic, B. J. Frey, and A. Kannan, ``Epitomic analysis of appearance and shape,'' in \textit{Proceedings of IEEE International Conference on Computer Vision (ICCV)}, pp. 34-43, 2003.

%\bibitem{Yang2008} J. Yang, J. Wright, T. Huang and Y. Ma, ``Image Super-resolution as Sparse Representation of Raw Image Patches,'' In Proceeding of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008.

\bibitem{NN}  W. T. Freeman, T. R. Jones, and E. C. Pasztor, ``Example-based super-resolution,'' \textit{IEEE Computer Graphics and Applications}, vol. 22, no. 2, pp. 56-65, 2002.

\bibitem{RG} K. I. Kim, and Y. Kwon, ``Single-image super-resolution using sparse regression and natural image prior,'' \textit{IEEE Transactions on Pattern Analysis and Machine Intelligence}, vol. 32, no. 6, pp. 1127-1133, 2010.

\bibitem{Lee2007} H. Lee, A. Battle, R. Raina and A. Y. Ng, ``Efficient sparse coding algorithms,'' in \textit{Proceedings of Neural Information Processing Systems (NIPS)}, pp. 801-808, 2007.

\bibitem{Yang2010} J. Yang, J. Wright, T. Huang, and Y. Ma, ``Image super-resolution via sparse representation,'' \textit{IEEE Transactions on Image Processing}, vol. 19, no. 11, pp. 2861-2873, 2010.

\bibitem{Huang2010} C. Yang, J. Huang, and M. Yang, ``Exploiting self-similarities for single frame super-resolution,'' in \textit{Proceedings of Asian Conference on Computer Vision (ACCV)}, pp. 1807-1818, 2010.

\bibitem{Dong2012} W. Dong, L.  Zhang, G. Shi and X. Li, ``Nonlocally centralized sparse representation for image restoration,'' \textit{IEEE Transactions on Image Processing}, vol. 22, no. 4, pp. 1620-1630, 2012.

\bibitem{Glasner2009} D. Glasner, S. Bagon and M. Irani, ``Super-resolution from a single image,'' in \textit{Proceeding of IEEE International Conference on Computer Vision (ICCV)}, pp. 349-356, 2009.

\bibitem{Mairal2009} J. Mairal, F. Bach, J. Ponce, G. Sapiro and A. Zisserman, ``Non-local sparse models for image restoration,'' in \textit{Proceedings of IEEE International Conference on Computer Vision (ICCV)}, pp. 2272-2279, 2009.

\bibitem{DD} P. Chatterjee, and P. Milanfar, ``Is denoising dead?'' \textit{IEEE Transactions on Image Processing}, vol. 19, no. 4, pp. 895-911, 2010.

\bibitem{Harold} H. Burger,  C. Schuler, and S. Harmeling,``Learning how to combine internal and external denoising methods,'' \textit{Lecture Notes in Computer Science: Pattern Recognition}, vol. 8142, pp. 121-130, 2013.

\bibitem{Zontak2011} M. Zontak, ``Internal statistics of a single natural image,'' in \textit{Proceedings of IEEE International Conference on Computer Vision (ICCV)}, pp. 977-984, 2011.

\bibitem{Irani} I. Mosseri, M. Zontak,  and M. Irani, ``Combining the power of internal and external denoising'', \textit{IEEE International Conference on Computational Photography (ICCP)}, pp. 1-9, 2013.

\bibitem{Dong} W. Dong, D. Zhang, G. Shi, and X. Wu, ``Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization'', \textit{IEEE Transactions on Image Processing}, vol. 20, no. 7, pp. 1838-1857, 2011.

\bibitem{Yang2013} J. Yang, Z. Lin, and S. Cohen, ``Fast image super-resolution based on in-place example regression,'' in \textit{Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, pp.1059-1066, 2013.

%\bibitem{WACV} Z. Wang, Z. Wang, S. Chang, J. Yang and T. Huang, ``A Joint Perspective Towards Image Super-Resolution: Unifying External- and Self-Examples,'' accepted by IEEE Winter conference on Applications of Computer Vision(WACV), 2014.

%\bibitem{Aharon2006A} M. Elad and M. Aharon, ``Image denoising via sparse and redundant representations over learned dictionaries,''
%   IEEE Transactions on Image Processing, vol. 15, no. 12, pp. 3736-3745, Dec. 2006.
%
%\bibitem{Aharon2006B} M. Aharon, M. Elad, and A. Bruckstein, ``K-SVD: An algorithm for designing overcomplete dictionaries for sparse
%   representation,'' IEEE Transactions on Signal Processing, vol. 54, no. 11, pp. 4311-4322, Nov. 2006.

\bibitem{Tang} C. Dong, C. C. Loy, K. He and X. Tang, ``Learning a deep convolutional network for image super-resolution'', in \textit{Proceedings of European Conference on Computer Vision (ECCV)}, pp.184-199, 2014.

\bibitem{A+} R. Tomofte, V. De Smet and L. Van Gool, ``A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution''.

\bibitem{WACV} Z. Wang, Z. Wang, S. Chang, J. Yang and T. S. Huang, ``A Joint Perspective Towards Image Super-Resolution: Unifying External- and Self-Examples,'' accepted by IEEE Winter conference on Applications of Computer Vision(WACV), 2014.

\bibitem{Ni09} K. Ni, A.~Kannan, A.~Criminisi, and J.~Winn, ``Epitomic location recognition,'' \textit{IEEE Transactions on Pattern Analysis and Machine Intelligence}, vol. 31, no. 12, pp. 2158-2167, 2009.

\bibitem{ChuYLCH10} X. Chu, S. Yan, L. Li, K. Chan, and T. Huang, ``Spatialized epitome and its applications'', \textit{IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, pp. 311-318, 2010.

\bibitem{SSIM} Wang, Zhou, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero P. Simoncelli. "Image quality assessment: from error visibility to structural similarity." Image Processing, IEEE Transactions on 13, no. 4 (2004): 600-612.

\bibitem{Yang14} Y. Yang, X.Chu, T. Ng, A. Chia, J. Yang, H. Jin and T. Huang, ``Epitomic image colorization'', in \textit{Proceedings of IEEE Conference on Acoustics, Speech, and Signal Processing (ICASSP)}, 2014.

\bibitem{Bertsekas1999} D. Bertsekas, \textit{Nonlinear Programming}, 2nd ed. Nashua, NH: Athena Scientific, 1999.

\bibitem{Bradley52rank} R. A. Bradley, and M. E. Terry,  ``Rank analysis of incomplete block designs: I. The method of paired comparisons,'' \textit{Biometrika}, pp. 324-345, 1952.

%\bibliography{thesisrefs}

\end{thebibliography}

\end{document}

1503.01138.extracted_bib
{Park2003 S. C. Park, M. K. Park, and K. G. Kang, ``Super-resolution image reconstruction: A technical overview,'' IEEE Signal Processing Magazine, vol. 20, no. 3, pp. 21-36, 2003. %
1503.01138.extracted_bib
{Morse2003 B. Morse, D. Schwartzwald, ``Image Magnification Using Level Set Reconstruction,'' In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 333-341, (2001)
1503.01138.extracted_bib
{Fattal2007 R. Fattal, ``Image upsampling via imposed edge statistics,'' ACM Transactions on Graphics, vol. 26, no. 3, pp. 95-102, 2007. %
1503.01138.extracted_bib
{Irani1991 M. Irani, S. Peleg, ``Improving Resolution by Image Registration,'' Computer Vision, Graphics and Image Processing, vol. 53, no. 3, pp. 231-239, 1991
1503.01138.extracted_bib
{Lin2004 Z. Lin, and H. Y. Shum, ``Fundamental limits of reconstruction-based superresolution algorithms under local translation,'' IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 1, pp. 83-97, 2004.
1503.01138.extracted_bib
{Yang2012 J. Yang, Z. Wang, Z. Lin, S. Chen, and T. Huang, ``Coupled Dictionary Training for Image Super-resolution,'' IEEE Transactions on Image Processing, vol. 21, no. 8, pp. 3467-3478, 2012.
1503.01138.extracted_bib
{Fattal2010 G. Freedman and R. Fattal, ``Image and video upscaling from local self-examples,'' ACM Transactions on Graphics, vol. 28, no. 3, 2010.
1503.01138.extracted_bib
{JojicFK03 N. Jojic, B. J. Frey, and A. Kannan, ``Epitomic analysis of appearance and shape,'' in Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 34-43, 2003. %
1503.01138.extracted_bib
{Yang2008 J. Yang, J. Wright, T. Huang and Y. Ma, ``Image Super-resolution as Sparse Representation of Raw Image Patches,'' In Proceeding of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008.
1503.01138.extracted_bib
{NN W. T. Freeman, T. R. Jones, and E. C. Pasztor, ``Example-based super-resolution,'' IEEE Computer Graphics and Applications, vol. 22, no. 2, pp. 56-65, 2002.
1503.01138.extracted_bib
{RG K. I. Kim, and Y. Kwon, ``Single-image super-resolution using sparse regression and natural image prior,'' IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 6, pp. 1127-1133, 2010.
1503.01138.extracted_bib
{Lee2007 H. Lee, A. Battle, R. Raina and A. Y. Ng, ``Efficient sparse coding algorithms,'' in Proceedings of Neural Information Processing Systems (NIPS), pp. 801-808, 2007.
1503.01138.extracted_bib
{Yang2010 J. Yang, J. Wright, T. Huang, and Y. Ma, ``Image super-resolution via sparse representation,'' IEEE Transactions on Image Processing, vol. 19, no. 11, pp. 2861-2873, 2010.
1503.01138.extracted_bib
{Huang2010 C. Yang, J. Huang, and M. Yang, ``Exploiting self-similarities for single frame super-resolution,'' in Proceedings of Asian Conference on Computer Vision (ACCV), pp. 1807-1818, 2010.
1503.01138.extracted_bib
{Dong2012 W. Dong, L. Zhang, G. Shi and X. Li, ``Nonlocally centralized sparse representation for image restoration,'' IEEE Transactions on Image Processing, vol. 22, no. 4, pp. 1620-1630, 2012.
1503.01138.extracted_bib
{Glasner2009 D. Glasner, S. Bagon and M. Irani, ``Super-resolution from a single image,'' in Proceeding of IEEE International Conference on Computer Vision (ICCV), pp. 349-356, 2009.
1503.01138.extracted_bib
{Mairal2009 J. Mairal, F. Bach, J. Ponce, G. Sapiro and A. Zisserman, ``Non-local sparse models for image restoration,'' in Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 2272-2279, 2009.
1503.01138.extracted_bib
{DD P. Chatterjee, and P. Milanfar, ``Is denoising dead?'' IEEE Transactions on Image Processing, vol. 19, no. 4, pp. 895-911, 2010.
1503.01138.extracted_bib
{Harold H. Burger, C. Schuler, and S. Harmeling,``Learning how to combine internal and external denoising methods,'' Lecture Notes in Computer Science: Pattern Recognition, vol. 8142, pp. 121-130, 2013.
1503.01138.extracted_bib
{Zontak2011 M. Zontak, ``Internal statistics of a single natural image,'' in Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 977-984, 2011.
1503.01138.extracted_bib
{Irani I. Mosseri, M. Zontak, and M. Irani, ``Combining the power of internal and external denoising'', IEEE International Conference on Computational Photography (ICCP), pp. 1-9, 2013.
1503.01138.extracted_bib
{Dong W. Dong, D. Zhang, G. Shi, and X. Wu, ``Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization'', IEEE Transactions on Image Processing, vol. 20, no. 7, pp. 1838-1857, 2011.
1503.01138.extracted_bib
{Yang2013 J. Yang, Z. Lin, and S. Cohen, ``Fast image super-resolution based on in-place example regression,'' in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.1059-1066, 2013. %
1503.01138.extracted_bib
{WACV Z. Wang, Z. Wang, S. Chang, J. Yang and T. Huang, ``A Joint Perspective Towards Image Super-Resolution: Unifying External- and Self-Examples,'' accepted by IEEE Winter conference on Applications of Computer Vision(WACV), 2014. %
1503.01138.extracted_bib
{Aharon2006A M. Elad and M. Aharon, ``Image denoising via sparse and redundant representations over learned dictionaries,'' % IEEE Transactions on Image Processing, vol. 15, no. 12, pp. 3736-3745, Dec. 2006. % %
1503.01138.extracted_bib
{Aharon2006B M. Aharon, M. Elad, and A. Bruckstein, ``K-SVD: An algorithm for designing overcomplete dictionaries for sparse % representation,'' IEEE Transactions on Signal Processing, vol. 54, no. 11, pp. 4311-4322, Nov. 2006.
1503.01138.extracted_bib
{Tang C. Dong, C. C. Loy, K. He and X. Tang, ``Learning a deep convolutional network for image super-resolution'', in Proceedings of European Conference on Computer Vision (ECCV), pp.184-199, 2014.
1503.01138.extracted_bib
{A+ R. Tomofte, V. De Smet and L. Van Gool, ``A+: Adjusted Anchored Neighborhood Regression for Fast Super-Resolution''.
1503.01138.extracted_bib
{WACV Z. Wang, Z. Wang, S. Chang, J. Yang and T. S. Huang, ``A Joint Perspective Towards Image Super-Resolution: Unifying External- and Self-Examples,'' accepted by IEEE Winter conference on Applications of Computer Vision(WACV), 2014.
1503.01138.extracted_bib
{Ni09 K. Ni, A.~Kannan, A.~Criminisi, and J.~Winn, ``Epitomic location recognition,'' IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 12, pp. 2158-2167, 2009.
1503.01138.extracted_bib
{ChuYLCH10 X. Chu, S. Yan, L. Li, K. Chan, and T. Huang, ``Spatialized epitome and its applications'', IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 311-318, 2010.
1503.01138.extracted_bib
{SSIM Wang, Zhou, Alan Conrad Bovik, Hamid Rahim Sheikh, and Eero P. Simoncelli. "Image quality assessment: from error visibility to structural similarity." Image Processing, IEEE Transactions on 13, no. 4 (2004): 600-612.
1503.01138.extracted_bib
{Yang14 Y. Yang, X.Chu, T. Ng, A. Chia, J. Yang, H. Jin and T. Huang, ``Epitomic image colorization'', in Proceedings of IEEE Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2014.
1503.01138.extracted_bib
{Bertsekas1999 D. Bertsekas, Nonlinear Programming, 2nd ed. Nashua, NH: Athena Scientific, 1999.
1503.01138.extracted_bib
{Bradley52rank R. A. Bradley, and M. E. Terry, ``Rank analysis of incomplete block designs: I. The method of paired comparisons,'' Biometrika, pp. 324-345, 1952. %\bibliography{thesisrefs