  There is an increasing interest in estimating expectations outside of the
classical inference framework, such as for models expressed as probabilistic
programs. Many of these contexts call for some form of nested inference to be
applied. In this paper, we analyse the behaviour of nested Monte Carlo (NMC)
schemes, for which classical convergence proofs are insufficient. We give
conditions under which NMC will converge, establish a rate of convergence, and
provide empirical data that suggests that this rate is observable in practice.
Finally, we prove that general-purpose nested inference schemes are inherently
biased. Our results serve to warn of the dangers associated with naive
composition of inference and models.
