  Full Bayesian posteriors are rarely analytically tractable, which is why
real-world Bayesian inference heavily relies on approximate techniques.
Approximations generally differ from the true posterior and require diagnostic
tools to assess whether the inference can still be trusted. We investigate a
new approach to diagnosing approximate inference: the approximation mismatch is
attributed to a change in the inductive bias by treating the approximations as
exact and reverse-engineering the corresponding prior. We show that the problem
is more complicated than it appears to be at first glance, because the solution
generally depends on the observation. By reframing the problem in terms of
incompatible conditional distributions we arrive at a natural solution: the
Gibbs prior. The resulting diagnostic is based on pseudo-Gibbs sampling, which
is widely applicable and easy to implement. We illustrate how the Gibbs prior
can be used to discover the inductive bias in a controlled Gaussian setting and
for a variety of Bayesian models and approximations.
