  Obtaining reliable, adaptive confidence sets for prediction functions
(hypotheses) is a central challenge in sequential decision-making tasks, such
as bandits and model-based reinforcement learning. These confidence sets
typically rely on prior assumptions on the hypothesis space, e.g., the known
kernel of a Reproducing Kernel Hilbert Space (RKHS). Hand-designing such
kernels is error prone, and misspecification may lead to poor or unsafe
performance. In this work, we propose to meta-learn a kernel from offline data
(Meta-KeL). For the case where the unknown kernel is a combination of known
base kernels, we develop an estimator based on structured sparsity. Under mild
conditions, we guarantee that our estimated RKHS yields valid confidence sets
that, with increasing amounts of offline data, become as tight as those given
the true unknown kernel. We demonstrate our approach on the kernelized bandit
problem (a.k.a.~Bayesian optimization), where we establish regret bounds
competitive with those given the true kernel. We also empirically evaluate the
effectiveness of our approach on a Bayesian optimization task.
