  In this work, we develop a novel input feature selection framework for
ReLU-based deep neural networks (DNNs), which builds upon a mixed-integer
optimization approach. While the method is generally applicable to various
classification tasks, we focus on finding input features for image
classification for clarity of presentation. The idea is to use a trained DNN,
or an ensemble of trained DNNs, to identify the salient input features. The
input feature selection is formulated as a sequence of mixed-integer linear
programming (MILP) problems that find sets of sparse inputs that maximize the
classification confidence of each category. These ''inverse'' problems are
regularized by the number of inputs selected for each category and by
distribution constraints. Numerical results on the well-known MNIST and
FashionMNIST datasets show that the proposed input feature selection allows us
to drastically reduce the size of the input to $\sim$15\% while maintaining a
good classification accuracy. This allows us to design DNNs with significantly
fewer connections, reducing computational effort and producing DNNs that are
more robust towards adversarial attacks.
