  We consider the problem of estimating a parameter associated to a Bayesian
inverse problem. Treating the unknown initial condition as a nuisance
parameter, typically one must resort to a numerical approximation of gradient
of the log-likelihood and also adopt a discretization of the problem in space
and/or time. We develop a new methodology to unbiasedly estimate the gradient
of the log-likelihood with respect to the unknown parameter, i.e. the
expectation of the estimate has no discretization bias. Such a property is not
only useful for estimation in terms of the original stochastic model of
interest, but can be used in stochastic gradient algorithms which benefit from
unbiased estimates. Under appropriate assumptions, we prove that our estimator
is not only unbiased but of finite variance. In addition, when implemented on a
single processor, we show that the cost to achieve a given level of error is
comparable to multilevel Monte Carlo methods, both practically and
theoretically. However, the new algorithm provides the possibility for parallel
computation on arbitrarily many processors without any loss of efficiency,
asymptotically. In practice, this means any precision can be achieved in a
fixed, finite constant time, provided that enough processors are available.
