  Scaling to larger systems, with current levels of reliability, requires
cost-effective methods to mitigate hardware failures. One of the main causes of
hardware failure is an uncorrected error in memory, which terminates the
current job and wastes all computation since the last checkpoint. This paper
presents the first adaptive method for triggering uncorrected error mitigation.
It uses a prediction approach that considers the likelihood of an uncorrected
error and its current potential cost. The method is based on reinforcement
learning, and the only user-defined parameters are the mitigation cost and
whether the job can be restarted from a mitigation point. We evaluate our
method using classical machine learning metrics together with a cost-benefit
analysis, which compares the cost of mitigation actions with the benefits from
mitigating some of the errors. On two years of production logs from the
MareNostrum supercomputer, our method reduces lost compute time by 54% compared
with no mitigation and is just 6% below the optimal Oracle method. All source
code is open source.
