  We provide a prescription to train optimal machine-learning-based event
selectors and categorizers that maximize the statistical significance of a
potential signal excess in high energy physics (HEP) experiments, as quantified
by any of six different performance measures. For analyses where the signal
search is performed in the distribution of some event variables, our
prescription ensures that only the information complementary to those event
variables is used in event selection and categorization. This eliminates a
major misalignment with the physics goals of the analysis (maximizing the
significance of an excess) that exists in the training of typical ML-based
event selectors and categorizers. In addition, this decorrelation of event
selectors from the relevant event variables prevents the background
distribution from becoming peaked in the signal region as a result of event
selection, thereby ameliorating the challenges imposed on signal searches by
systematic uncertainties. Our event selectors (categorizers) use the output of
machine-learning-based classifiers as input and apply optimal selection cutoffs
(categorization thresholds) that depend on the event variables being analyzed,
as opposed to flat cutoffs (thresholds). These optimal cutoffs and thresholds
are learned iteratively, using a novel approach with connections to Lloyd's
k-means clustering algorithm. We provide a public, Python 3 implementation of
our prescription called ThickBrick, along with usage examples.
