  The effect of the relative entropy asymmetry is analyzed in the empirical
risk minimization with relative entropy regularization (ERM-RER) problem. A
novel regularization is introduced, coined Type-II regularization, that allows
for solutions to the ERM-RER problem with a support that extends outside the
support of the reference measure. The solution to the new ERM-RER Type-II
problem is analytically characterized in terms of the Radon-Nikodym derivative
of the reference measure with respect to the solution. The analysis of the
solution unveils the following properties of relative entropy when it acts as a
regularizer in the ERM-RER problem: i) relative entropy forces the support of
the Type-II solution to collapse into the support of the reference measure,
which introduces a strong inductive bias that dominates the evidence provided
by the training data; ii) Type-II regularization is equivalent to classical
relative entropy regularization with an appropriate transformation of the
empirical risk function. Closed-form expressions of the expected empirical risk
as a function of the regularization parameters are provided.
