  Incorporating feature selection into a classification or regression method
often carries a number of advantages. In this paper we formalize feature
selection specifically from a discriminative perspective of improving
classification/regression accuracy. The feature selection method is developed
as an extension to the recently proposed maximum entropy discrimination (MED)
framework. We describe MED as a flexible (Bayesian) regularization approach
that subsumes, e.g., support vector classification, regression and exponential
family models. For brevity, we restrict ourselves primarily to feature
selection in the context of linear classification/regression methods and
demonstrate that the proposed approach indeed carries substantial improvements
in practice. Moreover, we discuss and develop various extensions of feature
selection, including the problem of dealing with example specific but
unobserved degrees of freedom -- alignments or invariants.
