  This paper presents an argument that certain AI safety measures, rather than
mitigating existential risk, may instead exacerbate it. Under certain key
assumptions - the inevitability of AI failure, the expected correlation between
an AI system's power at the point of failure and the severity of the resulting
harm, and the tendency of safety measures to enable AI systems to become more
powerful before failing - safety efforts have negative expected utility. The
paper examines three response strategies: Optimism, Mitigation, and Holism.
Each faces challenges stemming from intrinsic features of the AI safety
landscape that we term Bottlenecking, the Perfection Barrier, and Equilibrium
Fluctuation. The surprising robustness of the argument forces a re-examination
of core assumptions around AI safety and points to several avenues for further
research.
