  We demonstrate algorithm-based fault tolerance for silent, transient data
corruption in "black-box" preconditioners. We consider both additive Schwarz
domain decomposition with an ILU(k) subdomain solver, and algebraic multigrid,
both implemented in the Trilinos library. We evaluate faults that corrupt
preconditioner results in both single and multiple MPI ranks. We then analyze
how our approach behaves when then application is scaled. Our technique is
based on a Selective Reliability approach that performs most operations in an
unreliable mode, with only a few operations performed reliably. We also
investigate two responses to faults and discuss the performance overheads
imposed by each. For a non-symmetric problem solved using GMRES and ILU, we
show that at scale our fault tolerance approach incurs only 22% overhead for
the worst case. With detection techniques, we are able to reduce this overhead
to 1.8% in the worst case.
