  Diamond materials are central to an increasing range of advanced
technological demonstrations, from high power electronics, to nano-scale
quantum bio-imaging with unprecedented sensitivity. However, the full
exploitation of diamond for these applications is often limited by the
uncontrolled nature of the diamond material surface, which suffers from
Fermi-level pinning and hosts a significant density of electro-magnetic noise
sources. These issues occur despite the oxide-free and air-stable nature of the
diamond crystal surface, which should be an ideal candidate for
functionalization and chemical-engineering. In this work we reveal a family of
previously unidentified and near-ubiquitous primal surface defects which we
assign to differently reconstructed surface vacancies. The density of these
defects is quantified with X-ray absorption spectroscopy, their energy
structures are elucidated by ab initio calculations, and their effect on
near-surface quantum probes is measured directly. Subsequent ab-initio
calculation of band-bending from these defects suggest they are the source of
Fermi-level pinning at most diamond surfaces. Finally, an investigation is
conducted on a broad range of post-growth surface treatments and concludes that
none of them can reproducibly reduce this defect density below the
Fermi-pinning threshold, making this defect a prime candidate as the source for
decoherence-limiting noise in near-surface quantum probes.
