  As one of the central tasks in machine learning, regression finds lots of
applications in different fields. An existing common practice for solving
regression problems is the mean square error (MSE) minimization approach or its
regularized variants which require prior knowledge about the models. Recently,
Yi et al., proposed a mutual information based supervised learning framework
where they introduced a label entropy regularization which does not require any
prior knowledge. When applied to classification tasks and solved via a
stochastic gradient descent (SGD) optimization algorithm, their approach
achieved significant improvement over the commonly used cross entropy loss and
its variants. However, they did not provide a theoretical convergence analysis
of the SGD algorithm for the proposed formulation. Besides, applying the
framework to regression tasks is nontrivial due to the potentially infinite
support set of the label. In this paper, we investigate the regression under
the mutual information based supervised learning framework. We first argue that
the MSE minimization approach is equivalent to a conditional entropy learning
problem, and then propose a mutual information learning formulation for solving
regression problems by using a reparameterization technique. For the proposed
formulation, we give the convergence analysis of the SGD algorithm for solving
it in practice. Finally, we consider a multi-output regression data model where
we derive the generalization performance lower bound in terms of the mutual
information associated with the underlying data distribution. The result shows
that the high dimensionality can be a bless instead of a curse, which is
controlled by a threshold. We hope our work will serve as a good starting point
for further research on the mutual information based regression.
