  Gravitational lensing is a powerful tool for constraining substructure in the
mass distribution of galaxies, be it from the presence of dark matter sub-halos
or due to physical mechanisms affecting the baryons throughout galaxy
evolution. Such substructure is hard to model and is either ignored by
traditional, smooth modelling, approaches, or treated as well-localized massive
perturbers. In this work, we propose a deep learning approach to quantify the
statistical properties of such perturbations directly from images, where only
the extended lensed source features within a mask are considered, without the
need of any lens modelling. Our training data consist of mock lensed images
assuming perturbing Gaussian Random Fields permeating the smooth overall lens
potential, and, for the first time, using images of real galaxies as the lensed
source. We employ a novel deep neural network that can handle arbitrary
uncertainty intervals associated with the training dataset labels as input,
provides probability distributions as output, and adopts a composite loss
function. The method succeeds not only in accurately estimating the actual
parameter values, but also reduces the predicted confidence intervals by 10 per
cent in an unsupervised manner, i.e., without having access to the actual
ground truth values. Our results are invariant to the inherent degeneracy
between mass perturbations in the lens and complex brightness profiles for the
source. Hence, we can quantitatively and robustly quantify the smoothness of
the mass density of thousands of lenses, including confidence intervals, and
provide a consistent ranking for follow-up science.
