  The increasing adoption of Artificial Intelligence (AI) in engineering
problems calls for the development of calibration methods capable of offering
robust statistical reliability guarantees. The calibration of black box AI
models is carried out via the optimization of hyperparameters dictating
architecture, optimization, and/or inference configuration. Prior work has
introduced learn-then-test (LTT), a calibration procedure for hyperparameter
optimization (HPO) that provides statistical guarantees on average performance
measures. Recognizing the importance of controlling risk-aware objectives in
engineering contexts, this work introduces a variant of LTT that is designed to
provide statistical guarantees on quantiles of a risk measure. We illustrate
the practical advantages of this approach by applying the proposed algorithm to
a radio access scheduling problem.
