  We study the out-of-sample properties of robust empirical optimization
problems with smooth $\phi$-divergence penalties and smooth concave objective
functions, and develop a theory for data-driven calibration of the non-negative
"robustness parameter" $\delta$ that controls the size of the deviations from
the nominal model. Building on the intuition that robust optimization reduces
the sensitivity of the expected reward to errors in the model by controlling
the spread of the reward distribution, we show that the first-order benefit of
``little bit of robustness" (i.e., $\delta$ small, positive) is a significant
reduction in the variance of the out-of-sample reward while the corresponding
impact on the mean is almost an order of magnitude smaller. One implication is
that substantial variance (sensitivity) reduction is possible at little cost if
the robustness parameter is properly calibrated. To this end, we introduce the
notion of a robust mean-variance frontier to select the robustness parameter
and show that it can be approximated using resampling methods like the
bootstrap. Our examples show that robust solutions resulting from "open loop"
calibration methods (e.g., selecting a $90\%$ confidence level regardless of
the data and objective function) can be very conservative out-of-sample, while
those corresponding to the robustness parameter that optimizes an estimate of
the out-of-sample expected reward (e.g., via the bootstrap) with no regard for
the variance are often insufficiently robust.
