  Prediction-powered inference (PPI) is a method that improves statistical
estimates based on limited human-labeled data. PPI achieves this by combining
small amounts of human-labeled data with larger amounts of data labeled by a
reasonably accurate -- but potentially biased -- automatic system, in a way
that results in tighter confidence intervals for certain parameters of interest
(e.g., the mean performance of a language model). In this paper, we propose a
method called Stratified Prediction-Powered Inference (StratPPI), in which we
show that the basic PPI estimates can be considerably improved by employing
simple data stratification strategies. Without making any assumptions on the
underlying automatic labeling system or data distribution, we derive an
algorithm for computing provably valid confidence intervals for population
parameters (such as averages) that is based on stratified sampling. In
particular, we show both theoretically and empirically that, with appropriate
choices of stratification and sample allocation, our approach can provide
substantially tighter confidence intervals than unstratified approaches.
Specifically, StratPPI is expected to improve in cases where the performance of
the autorater varies across different conditional distributions of the target
data.
