  When training a machine learning classifier on data where one of the classes
is intrinsically rare, the classifier will often assign too few sources to the
rare class. To address this, it is common to up-weight the examples of the rare
class to ensure it isn't ignored. It is also a frequent practice to train on
restricted data where the balance of source types is closer to equal for the
same reason. Here we show that these practices can bias the model toward
over-assigning sources to the rare class. We also explore how to detect when
training data bias has had a statistically significant impact on the trained
model's predictions, and how to reduce the bias's impact. While the magnitude
of the impact of the techniques developed here will vary with the details of
the application, for most cases it should be modest. They are, however,
universally applicable to every time a machine learning classification model is
used, making them analogous to Bessel's correction to the sample variance.
