  We operate a fiberbased cavity with an inserted diamond membrane containing
ensembles of silicon vacancy centers (SiV$^-$) at cryogenic temperatures $
\geq4~$K. The setup, sample fabrication and spectroscopic characterization is
described, together with a demonstration of the cavity influence by the Purcell
effect. This paves the way towards solid state qubits coupled to optical
interfaces as long-lived quantum memories.
