  This paper proposes a new method for preventing unsafe or otherwise low
quality large language model (LLM) outputs, by leveraging the stochasticity of
LLMs. We propose a system whereby LLM checkers vote on the acceptability of a
generated output, regenerating it if a threshold of disapproval is reached,
until sufficient checkers approve. We further propose estimators for cost and
failure rate, and based on those estimators and experimental data tailored to
the application, we propose an algorithm that achieves a desired failure rate
at the least possible cost. We demonstrate that, under these models, failure
rate decreases exponentially as a function of cost when voter count and
threshold are chosen according to the algorithm, and that the models reasonably
estimate the actual performance of such a system in action, even with limited
data.
