  It is challenging to guide neural network (NN) learning with prior knowledge.
In contrast, many known properties, such as spatial smoothness or seasonality,
are straightforward to model by choosing an appropriate kernel in a Gaussian
process (GP). Many deep learning applications could be enhanced by modeling
such known properties. For example, convolutional neural networks (CNNs) are
frequently used in remote sensing, which is subject to strong seasonal effects.
We propose to blend the strengths of deep learning and the clear modeling
capabilities of GPs by using a composite kernel that combines a kernel
implicitly defined by a neural network with a second kernel function chosen to
model known properties (e.g., seasonality). We implement this idea by combining
a deep network and an efficient mapping based on the Nystrom approximation,
which we call Implicit Composite Kernel (ICK). We then adopt a
sample-then-optimize approach to approximate the full GP posterior
distribution. We demonstrate that ICK has superior performance and flexibility
on both synthetic and real-world data sets. We believe that ICK framework can
be used to include prior information into neural networks in many applications.
