  Deep learning methods, in particular trained Convolutional Neural Networks
(CNNs) have recently been shown to produce compelling state-of-the-art results
for single image Super-Resolution (SR). Invariably, a CNN is learned to map the
low resolution (LR) image to its corresponding high resolution (HR) version in
the spatial domain. Aiming for faster inference and more efficient solutions
than solving the SR problem in the spatial domain, we propose a novel network
structure for learning the SR mapping function in an image transform domain,
specifically the Discrete Cosine Transform (DCT). As a first contribution, we
show that DCT can be integrated into the network structure as a Convolutional
DCT (CDCT) layer. We further extend the network to allow the CDCT layer to
become trainable (i.e. optimizable). Because this layer represents an image
transform, we enforce pairwise orthogonality constraints on the individual
basis functions/filters. This Orthogonally Regularized Deep SR network (ORDSR)
simplifies the SR task by taking advantage of image transform domain while
adapting the design of transform basis to the training image set.
