  With the rise of stochastic generative models in robot policy learning,
end-to-end visuomotor policies are increasingly successful at solving complex
tasks by learning from human demonstrations. Nevertheless, since real-world
evaluation costs afford users only a small number of policy rollouts, it
remains a challenge to accurately gauge the performance of such policies. This
is exacerbated by distribution shifts causing unpredictable changes in
performance during deployment. To rigorously evaluate behavior cloning
policies, we present a framework that provides a tight lower-bound on robot
performance in an arbitrary environment, using a minimal number of experimental
policy rollouts. Notably, by applying the standard stochastic ordering to robot
performance distributions, we provide a worst-case bound on the entire
distribution of performance (via bounds on the cumulative distribution
function) for a given task. We build upon established statistical results to
ensure that the bounds hold with a user-specified confidence level and
tightness, and are constructed from as few policy rollouts as possible. In
experiments we evaluate policies for visuomotor manipulation in both simulation
and hardware. Specifically, we (i) empirically validate the guarantees of the
bounds in simulated manipulation settings, (ii) find the degree to which a
learned policy deployed on hardware generalizes to new real-world environments,
and (iii) rigorously compare two policies tested in out-of-distribution
settings. Our experimental data, code, and implementation of confidence bounds
are open-source.
