  Boosting methods combine a set of moderately accurate weaklearners to form a
highly accurate predictor. Despite the practical importance of multi-class
boosting, it has received far less attention than its binary counterpart. In
this work, we propose a fully-corrective multi-class boosting formulation which
directly solves the multi-class problem without dividing it into multiple
binary classification problems. In contrast, most previous multi-class boosting
algorithms decompose a multi-boost problem into multiple binary boosting
problems. By explicitly deriving the Lagrange dual of the primal optimization
problem, we are able to construct a column generation-based fully-corrective
approach to boosting which directly optimizes multi-class classification
performance. The new approach not only updates all weak learners' coefficients
at every iteration, but does so in a manner flexible enough to accommodate
various loss functions and regularizations. For example, it enables us to
introduce structural sparsity through mixed-norm regularization to promote
group sparsity and feature sharing. Boosting with shared features is
particularly beneficial in complex prediction problems where features can be
expensive to compute. Our experiments on various data sets demonstrate that our
direct multi-class boosting generalizes as well as, or better than, a range of
competing multi-class boosting methods. The end result is a highly effective
and compact ensemble classifier which can be trained in a distributed fashion.
