  Gaussian Markov random fields (GMRFs) are probabilistic graphical models
widely used in spatial statistics and related fields to model dependencies over
spatial structures. We establish a formal connection between GMRFs and
convolutional neural networks (CNNs). Common GMRFs are special cases of a
generative model where the inverse mapping from data to latent variables is
given by a 1-layer linear CNN. This connection allows us to generalize GMRFs to
multi-layer CNN architectures, effectively increasing the order of the
corresponding GMRF in a way which has favorable computational scaling. We
describe how well-established tools, such as autodiff and variational
inference, can be used for simple and efficient inference and learning of the
deep GMRF. We demonstrate the flexibility of the proposed model and show that
it outperforms the state-of-the-art on a dataset of satellite temperatures, in
terms of prediction and predictive uncertainty.
