  We consider posterior sampling in the very common Bayesian hierarchical model
in which observed data depends on high-dimensional latent variables that, in
turn, depend on relatively few hyperparameters. When the full conditional over
the latent variables has a known form, the marginal posterior distribution over
hyperparameters is accessible and can be sampled using a Markov chain Monte
Carlo (MCMC) method on a low-dimensional parameter space. This may improve
computational efficiency over standard Gibbs sampling since computation is not
over the high-dimensional space of latent variables and correlations between
hyperparameters and latent variables become irrelevant. When the marginal
posterior over hyperparameters depends on a fixed-dimensional sufficient
statistic, precomputation of the sufficient statistic renders the cost of the
low-dimensional MCMC independent of data size. Then, when the hyperparameters
are the primary variables of interest, inference may be performed in big-data
settings at modest cost. Moreover, since the form of the full conditional for
the latent variables does not depend on the form of the hyperprior
distribution, the method imposes no restriction on the hyperprior, unlike Gibbs
sampling that typically requires conjugate distributions. We demonstrate these
efficiency gains in four computed examples.
