  In many important machine learning applications, the standard assumption of
having a globally Lipschitz continuous gradient may fail to hold. This paper
delves into a more general $(L_0, L_1)$-smoothness setting, which gains
particular significance within the realms of deep neural networks and
distributionally robust optimization (DRO). We demonstrate the significant
advantage of trust region methods for stochastic nonconvex optimization under
such generalized smoothness assumption. We show that first-order trust region
methods can recover the normalized and clipped stochastic gradient as special
cases and then provide a unified analysis to show their convergence to
first-order stationary conditions. Motivated by the important application of
DRO, we propose a generalized high-order smoothness condition, under which
second-order trust region methods can achieve a complexity of
$\mathcal{O}(\epsilon^{-3.5})$ for convergence to second-order stationary
points. By incorporating variance reduction, the second-order trust region
method obtains an even better complexity of $\mathcal{O}(\epsilon^{-3})$,
matching the optimal bound for standard smooth optimization. To our best
knowledge, this is the first work to show convergence beyond the first-order
stationary condition for generalized smooth optimization. Preliminary
experiments show that our proposed algorithms perform favorably compared with
existing methods.
