  Stochastic optimisation algorithms are the de facto standard for machine
learning with large amounts of data. Handling only a subset of available data
in each optimisation step dramatically reduces the per-iteration computational
costs, while still ensuring significant progress towards the solution. Driven
by the need to solve large-scale optimisation problems as efficiently as
possible, the last decade has witnessed an explosion of research in this area.
Leveraging the parallels between machine learning and inverse problems has
allowed harnessing the power of this research wave for solving inverse
problems. In this survey, we provide a comprehensive account of the
state-of-the-art in stochastic optimisation from the viewpoint of inverse
problems. We present algorithms with diverse modalities of problem
randomisation and discuss the roles of variance reduction, acceleration,
higher-order methods, and other algorithmic modifications, and compare
theoretical results with practical behaviour. We focus on the potential and the
challenges for stochastic optimisation that are unique to inverse imaging
problems and are not commonly encountered in machine learning. We conclude the
survey with illustrative examples from imaging problems to examine the
advantages and disadvantages that this new generation of algorithms bring to
the field of inverse problems.
