  We present fast classification techniques for sparse generalized linear and
additive models. These techniques can handle thousands of features and
thousands of observations in minutes, even in the presence of many highly
correlated features. For fast sparse logistic regression, our computational
speed-up over other best-subset search techniques owes to linear and quadratic
surrogate cuts for the logistic loss that allow us to efficiently screen
features for elimination, as well as use of a priority queue that favors a more
uniform exploration of features. As an alternative to the logistic loss, we
propose the exponential loss, which permits an analytical solution to the line
search at each iteration. Our algorithms are generally 2 to 5 times faster than
previous approaches. They produce interpretable models that have accuracy
comparable to black box models on challenging datasets.
