  Perception systems operate as a subcomponent of the general autonomy stack,
and perception system designers often need to optimize performance
characteristics while maintaining safety with respect to the overall
closed-loop system. For this reason, it is useful to distill high-level safety
requirements into component-level requirements on the perception system. In
this work, we focus on efficiently determining sets of safe perception system
performance characteristics given a black-box simulator of the
fully-integrated, closed-loop system. We combine the advantages of common
black-box estimation techniques such as Gaussian processes and threshold
bandits to develop a new estimation method, which we call smoothing bandits. We
demonstrate our method on a vision-based aircraft collision avoidance problem
and show improvements in terms of both accuracy and efficiency over the
Gaussian process and threshold bandit baselines.
