  Generative models, like large language models, are becoming increasingly
relevant in our daily lives, yet a theoretical framework to assess their
generalization behavior and uncertainty does not exist. Particularly, the
problem of uncertainty estimation is commonly solved in an ad-hoc and
task-dependent manner. For example, natural language approaches cannot be
transferred to image generation. In this paper, we introduce the first
bias-variance-covariance decomposition for kernel scores. This decomposition
represents a theoretical framework from which we derive a kernel-based variance
and entropy for uncertainty estimation. We propose unbiased and consistent
estimators for each quantity which only require generated samples but not the
underlying model itself. Based on the wide applicability of kernels, we
demonstrate our framework via generalization and uncertainty experiments for
image, audio, and language generation. Specifically, kernel entropy for
uncertainty estimation is more predictive of performance on CoQA and TriviaQA
question answering datasets than existing baselines and can also be applied to
closed-source models.
