  Deep learning has been widely used for solving image reconstruction tasks but
its deployability has been held back due to the shortage of high-quality
training data. Unsupervised learning methods, such as the deep image prior
(DIP), naturally fill this gap, but bring a host of new issues: the
susceptibility to overfitting due to a lack of robust early stopping strategies
and unstable convergence. We present a novel approach to tackle these issues by
restricting DIP optimisation to a sparse linear subspace of its parameters,
employing a synergy of dimensionality reduction techniques and second order
optimisation methods. The low-dimensionality of the subspace reduces DIP's
tendency to fit noise and allows the use of stable second order optimisation
methods, e.g., natural gradient descent or L-BFGS. Experiments across both
image restoration and tomographic tasks of different geometry and ill-posedness
show that second order optimisation within a low-dimensional subspace is
favourable in terms of optimisation stability to reconstruction fidelity
trade-off.
