  In astronomy, we frequently face the decision problem: does this data contain
a signal? Typically, a statistical approach is used, which requires a
threshold. The choice of threshold presents a common challenge in settings
where signals and noise must be delineated, but their distributions overlap.
Gravitational-wave astronomy, which has gone from the first discovery to
catalogues of hundreds of events in less than a decade, presents a fascinating
case study. For signals from colliding compact objects, the field has evolved
from a frequentist to a Bayesian methodology. However, the issue of choosing a
threshold and validating noise contamination in a catalogue persists. Confusion
and debate often arise due to the misapplication of statistical concepts, the
complicated nature of the detection statistics, and the inclusion of
astrophysical background models. We introduce Conformal Prediction (CP), a
framework developed in Machine Learning to provide distribution-free
uncertainty quantification to point predictors. We show that CP can be viewed
as an extension of the traditional statistical frameworks whereby thresholds
are calibrated such that the uncertainty intervals are statistically rigorous
and the error rate can be validated. Moreover, we discuss how CP offers a
framework to optimally build a meta-pipeline combining the outputs from
multiple independent searches. We introduce CP with a toy cosmic-ray detector,
which captures the salient features of most astrophysical search problems and
allows us to demonstrate the features of CP in a simple context. We then apply
the approach to a recent gravitational-wave Mock Data Challenge using multiple
search algorithms for compact binary coalescence signals in interferometric
gravitational-wave data. Finally, we conclude with a discussion on the future
potential of the method for gravitational-wave astronomy.
