  Exponential models of distributions are widely used in machine learning for
classiffication and modelling. It is well known that they can be interpreted as
maximum entropy models under empirical expectation constraints. In this work,
we argue that for classiffication tasks, mutual information is a more suitable
information theoretic measure to be optimized. We show how the principle of
minimum mutual information generalizes that of maximum entropy, and provides a
comprehensive framework for building discriminative classiffiers. A game
theoretic interpretation of our approach is then given, and several
generalization bounds provided. We present iterative algorithms for solving the
minimum information problem and its convex dual, and demonstrate their
performance on various classiffication tasks. The results show that minimum
information classiffiers outperform the corresponding maximum entropy models.
