  This paper studies the problem of learning the conditional distribution of a
high-dimensional output given an input, where the output and input may belong
to two different domains, e.g., the output is a photo image and the input is a
sketch image. We solve this problem by cooperative training of a fast thinking
initializer and slow thinking solver. The initializer generates the output
directly by a non-linear transformation of the input as well as a noise vector
that accounts for latent variability in the output. The slow thinking solver
learns an objective function in the form of a conditional energy function, so
that the output can be generated by optimizing the objective function, or more
rigorously by sampling from the conditional energy-based model. We propose to
learn the two models jointly, where the fast thinking initializer serves to
initialize the sampling of the slow thinking solver, and the solver refines the
initial output by an iterative algorithm. The solver learns from the difference
between the refined output and the observed output, while the initializer
learns from how the solver refines its initial output. We demonstrate the
effectiveness of the proposed method on various conditional learning tasks,
e.g., class-to-image generation, image-to-image translation, and image
recovery. The advantage of our method over GAN-based methods is that our method
is equipped with a slow thinking process that refines the solution guided by a
learned objective function.
