  We consider the stochastic gradient method with random reshuffling
($\mathsf{RR}$) for tackling smooth nonconvex optimization problems.
$\mathsf{RR}$ finds broad applications in practice, notably in training neural
networks. In this work, we first investigate the concentration property of
$\mathsf{RR}$'s sampling procedure and establish a new high probability sample
complexity guarantee for driving the gradient (without expectation) below
$\varepsilon$, which effectively characterizes the efficiency of a single
$\mathsf{RR}$ execution. Our derived complexity matches the best existing
in-expectation one up to a logarithmic term while imposing no additional
assumptions nor changing $\mathsf{RR}$'s updating rule. Furthermore, by
leveraging our derived high probability descent property and bound on the
stochastic error, we propose a simple and computable stopping criterion for
$\mathsf{RR}$ (denoted as $\mathsf{RR}$-$\mathsf{sc}$). This criterion is
guaranteed to be triggered after a finite number of iterations, and then
$\mathsf{RR}$-$\mathsf{sc}$ returns an iterate with its gradient below
$\varepsilon$ with high probability. Moreover, building on the proposed
stopping criterion, we design a perturbed random reshuffling method
($\mathsf{p}$-$\mathsf{RR}$) that involves an additional randomized
perturbation procedure near stationary points. We derive that
$\mathsf{p}$-$\mathsf{RR}$ provably escapes strict saddle points and
efficiently returns a second-order stationary point with high probability,
without making any sub-Gaussian tail-type assumptions on the stochastic
gradient errors. Finally, we conduct numerical experiments on neural network
training to support our theoretical findings.
