  The negatively-charged nitrogen vacancy (NV$^-$) centre in diamond is a
remarkable optical quantum sensor for a range of applications including,
nanoscale thermometry, magnetometry, single photon generation, quantum
computing, and communication. However, to date the performance of these
techniques using NV$^-$ centres has been limited by the thermally-induced
spectral wandering of NV$^-$ centre photoluminescence due to detrimental
photothermal heating. Here we demonstrate that solid-state laser refrigeration
can be used to enable rapid (ms) optical temperature control of nitrogen
vacancy doped nanodiamond (NV$^-$:ND) quantum sensors in both atmospheric and
\textit{in vacuo} conditions. Nanodiamonds are attached to ceramic
microcrystals including 10\% ytterbium doped yttrium lithium fluoride
(Yb:LiYF$_4$) and sodium yttrium fluoride (Yb:NaYF$_4$) by van der Waals
bonding. The fluoride crystals were cooled through the efficient emission of
upconverted infrared photons excited by a focused 1020 nm laser beam. Heat
transfer to the ceramic microcrystals cooled the adjacent NV$^-$:NDs by 10 and
27 K at atmospheric pressure and $\sim$10$^{-3}$ Torr, respectively. The
temperature of the NV$^-$:NDs was measured using both Debye-Waller factor (DWF)
thermometry and optically detected magnetic resonance (ODMR), which agree with
the temperature of the laser cooled ceramic microcrystal. Stabilization of
thermally-induced spectral wandering of the NV$^{-}$ zero-phonon-line (ZPL) is
achieved by modulating the 1020 nm laser irradiance. The demonstrated cooling
of NV$^-$:NDs using an optically cooled microcrystal opens up new possibilities
for rapid feedback-controlled cooling of a wide range of nanoscale quantum
materials.
