  We identify a major source of quantum decoherence in three-dimensional
superconducting radio-frequency (SRF) resonators and two-dimensional transmon
qubits composed of oxidized niobium: oxygen vacancies in the niobium pentoxide
which drive two-level system (TLS) losses. By probing the effect of sequential
\textit{in situ} vacuum baking treatments on the RF performance of bulk Nb SRF
resonators and on the oxide structure of a representative Nb sample using
time-of-flight secondary ion mass spectrometry (ToF-SIMS), we find a
non-monotonic evolution of cavity quality factor $Q_0$ which correlates with
the interplay of Nb\textsubscript{2}O\textsubscript{5} vacancy generation and
oxide thickness reduction. We localize this effect to the oxide itself and
present the insignificant role of diffused interstitial oxygen in the
underlying Nb by regrowing a new oxide \textit{via} wet oxidation which reveals
a mitigation of aggravated TLS losses. We hypothesize that such vacancies in
the pentoxide serve as magnetic impurities and are a source of TLS-driven RF
loss.
