  The search for extraterrestrial (alien) life is one of the greatest
scientific quests yet raises fundamental questions about just what we should be
looking for and how. We approach alien hunting from the perspective of an
experimenter engaging in binary classification with some true and confounding
positive probability (TPP and CPP). We derive the Bayes factor in such a
framework between two competing hypotheses, which we use to classify
experiments as either impotent, imperfect or ideal. Similarly, the experimenter
can be classified as dogmatic, biased or agnostic. We show how the unbounded
explanatory and evasion capability of aliens poses fundamental problems to
experiments directly seeking aliens. Instead, we advocate framing the
experiments as looking for that outside of known processes, which means the
hypotheses we test do not directly concern aliens per se. To connect back to
aliens requires a second level of model selection, for which we derive the
final odds ratio in a Bayesian framework. This reveals that it is fundamentally
impossible to ever establish alien life at some threshold odds ratio,
$\mathcal{O}_{\mathrm{crit}}$, unless we deem the prior probability that some
as-yet-undiscovered natural process could explain the event is less than
$(1+\mathcal{O}_{\mathrm{crit}})^{-1}$. This elucidates how alien hunters need
to carefully consider the challenging problem of how probable unknown unknowns
are, such as new physics or chemistry, and how it is arguably most fruitful to
focus on experiments for which our domain knowledge is thought to be
asymptotically complete.
