  We consider a typical learning problem of point estimations for modeling of
nonlinear functions or dynamical systems in which generalization, i.e.,
verifying a given learned model, can be embedded as an integral part of the
learning process or dynamics. In particular, we consider an empirical risk
minimization based learning problem that exploits gradient methods from
continuous-time perspective with small random perturbations, which is guided by
the training dataset loss. Here, we provide an asymptotic probability estimate
in the small noise limit based-on the Freidlin-Wentzell theory of large
deviations, when the sample path of the random process corresponding to the
randomly perturbed gradient dynamical system hits a certain target set, i.e., a
rare event, when the latter is specified by the testing dataset loss landscape.
Interestingly, the proposed framework can be viewed as one way of improving
generalization and robustness in learning problems that provides new insights
leading to optimal point estimates which is guided by training data loss,
while, at the same time, the learning dynamics has an access to the testing
dataset loss landscape in some form of future achievable or anticipated target
goal. Moreover, as a by-product, we establish a connection with optimal control
problem, where the target set, i.e., the rare event, is considered as the
desired outcome or achievable target goal for a certain optimal control
problem, for which we also provide a verification result reinforcing the
rationale behind the proposed framework. Finally, we present a computational
algorithm that solves the corresponding variational problem leading to an
optimal point estimates and, as part of this work, we also present some
numerical results for a typical case of nonlinear regression problem.
