  Marginally outer trapped surfaces (MOTSs, or marginal surfaces in short) are
routinely used in numerical simulations of black hole spacetimes. They are an
invaluable tool for locating and characterizing black holes quasi-locally in
real time while the simulation is ongoing. It is often believed that a MOTS can
behave unpredictably under time evolution; an existing MOTS can disappear, and
a new one can appear without any apparent reason. In this paper we show that in
fact the behavior of a MOTS is perfectly predictable and its behavior is
dictated by a single real parameter, the \emph{stability parameter}, which can
be monitored during the course of a numerical simulation. We demonstrate the
utility of the stability parameter to fully understand the variety of marginal
surfaces that can be present in binary black hole initial data. We also develop
a new horizon finder capable of locating very highly distorted marginal
surfaces and we show that even in these cases, the stability parameter
perfectly predicts the existence and stability of marginal surfaces.
