  Using images from the Helioseismic and Magnetic Imager aboard the
\textit{Solar Dynamics Observatory} (SDO/HMI), we extract the radial-velocity
(RV) signal arising from the suppression of convective blue-shift and from
bright faculae and dark sunspots transiting the rotating solar disc. We remove
these rotationally modulated magnetic-activity contributions from simultaneous
radial velocities observed by the HARPS-N solar feed to produce a
radial-velocity time series arising from the magnetically quiet solar surface
(the 'inactive-region radial velocities'). We find that the level of
variability in the inactive-region radial velocities remains constant over the
almost 7 year baseline and shows no correlation with well-known activity
indicators. With an RMS of roughly 1 m/s, the inactive-region radial-velocity
time series dominates the total RV variability budget during the decline of
solar cycle 24. Finally, we compare the variability amplitude and timescale of
the inactive-region radial velocities with simulations of supergranulation. We
find consistency between the inactive-region radial-velocity and simulated time
series, indicating that supergranulation is a significant contribution to the
overall solar radial velocity variability, and may be the main source of
variability towards solar minimum. This work highlights supergranulation as a
key barrier to detecting Earth twins.
