  Learning the kernel parameters for Gaussian processes is often the
computational bottleneck in applications such as online learning, Bayesian
optimization, or active learning. Amortizing parameter inference over different
datasets is a promising approach to dramatically speed up training time.
However, existing methods restrict the amortized inference procedure to a fixed
kernel structure. The amortization network must be redesigned manually and
trained again in case a different kernel is employed, which leads to a large
overhead in design time and training time. We propose amortizing kernel
parameter inference over a complete kernel-structure-family rather than a fixed
kernel structure. We do that via defining an amortization network over pairs of
datasets and kernel structures. This enables fast kernel inference for each
element in the kernel family without retraining the amortization network. As a
by-product, our amortization network is able to do fast ensembling over kernel
structures. In our experiments, we show drastically reduced inference time
combined with competitive test performance for a large set of kernels and
datasets.
