  Knowledge of the nitrogen-vacancy center formation kinetics in diamond is
critical to engineering sensors and quantum information devices based on this
defect. Here we utilize the longitudinal tracking of single NV centers to
elucidate NV defect kinetics during high-temperature annealing from 800-1100
$^\circ$C in high-purity chemical-vapor-deposition diamond. We observe three
phenomena which can coexist: NV formation, NV quenching, and NV orientation
changes. Of relevance to NV-based applications, a 6 to 24-fold enhancement in
the NV density, in the absence of sample irradiation, is observed by annealing
at 980 $^\circ$C, and NV orientation changes are observed at 1050 $^\circ$C.
With respect to the fundamental understanding of defect kinetics in ultra-pure
diamond, our results indicate a significant vacancy source can be activated for
NV creation between 950-980 $^\circ$C and suggests that native hydrogen from
NVH$_y$ complexes plays a dominant role in NV quenching, in agreement with
recent {\it ab initio} calculations. Finally, the direct observation of
orientation changes allows us to estimate an NV diffusion barrier of 5.1~eV.
