  We introduce the convolutional spectral kernel (CSK), a novel family of
non-stationary, nonparametric covariance kernels for Gaussian process (GP)
models, derived from the convolution between two imaginary radial basis
functions. We present a principled framework to interpret CSK, as well as other
deep probabilistic models, using approximated Fourier transform, yielding a
concise representation of input-frequency spectrogram. Observing through the
lens of the spectrogram, we provide insight on the interpretability of deep
models. We then infer the functional hyperparameters using scalable variational
and MCMC methods. On small- and medium-sized spatiotemporal datasets, we
demonstrate improved generalization of GP models when equipped with CSK, and
their capability to extract non-stationary periodic patterns.
