  Stemming from information-theoretic learning, the correntropy criterion and
its applications to machine learning tasks have been extensively explored and
studied. Its application to regression problems leads to the robustness
enhanced regression paradigm -- namely, correntropy based regression. Having
drawn a great variety of successful real-world applications, its theoretical
properties have also been investigated recently in a series of studies from a
statistical learning viewpoint. The resulting big picture is that correntropy
based regression regresses towards the conditional mode function or the
conditional mean function robustly under certain conditions. Continuing this
trend and going further, in the present study, we report some new insights into
this problem. First, we show that under the additive noise regression model,
such a regression paradigm can be deduced from minimum distance estimation,
implying that the resulting estimator is essentially a minimum distance
estimator and thus possesses robustness properties. Second, we show that the
regression paradigm, in fact, provides a unified approach to regression
problems in that it approaches the conditional mean, the conditional mode, as
well as the conditional median functions under certain conditions. Third, we
present some new results when it is utilized to learn the conditional mean
function by developing its error bounds and exponential convergence rates under
conditional $(1+\epsilon)$-moment assumptions. The saturation effect on the
established convergence rates, which was observed under $(1+\epsilon)$-moment
assumptions, still occurs, indicating the inherent bias of the regression
estimator. These novel insights deepen our understanding of correntropy based
regression, help cement the theoretic correntropy framework, and also enable us
to investigate learning schemes induced by general bounded nonconvex loss
functions.
