  The problem of best subset selection in linear regression is considered with
the aim to find a fixed size subset of features that best fits the response.
This is particularly challenging when the total available number of features is
very large compared to the number of data samples. Existing optimal methods for
solving this problem tend to be slow while fast methods tend to have low
accuracy. Ideally, new methods perform best subset selection faster than
existing optimal methods but with comparable accuracy, or, being more accurate
than methods of comparable computational speed. Here, we propose a novel
continuous optimization method that identifies a subset solution path, a small
set of models of varying size, that consists of candidates for the single best
subset of features, that is optimal in a specific sense in linear regression.
Our method turns out to be fast, making the best subset selection possible when
the number of features is well in excess of thousands. Because of the
outstanding overall performance, framing the best subset selection challenge as
a continuous optimization problem opens new research directions for feature
extraction for a large variety of regression models.
