  We consider high-dimensional binary classification by sparse logistic
regression. We propose a model/feature selection procedure based on penalized
maximum likelihood with a complexity penalty on the model size and derive the
non-asymptotic bounds for the resulting misclassification excess risk. The
bounds can be reduced under the additional low-noise condition. The proposed
complexity penalty is remarkably related to the VC-dimension of a set of sparse
linear classifiers. Implementation of any complexity penalty-based criterion,
however, requires a combinatorial search over all possible models. To find a
model selection procedure computationally feasible for high-dimensional data,
we extend the Slope estimator for logistic regression and show that under an
additional weighted restricted eigenvalue condition it is rate-optimal in the
minimax sense.
