  We study the problem of feature selection in general machine learning (ML)
context, which is one of the most critical subjects in the field. Although,
there exist many feature selection methods, however, these methods face
challenges such as scalability, managing high-dimensional data, dealing with
correlated features, adapting to variable feature importance, and integrating
domain knowledge. To this end, we introduce the "Adaptive Feature Selection
with Binary Masking" (AFS-BM) which remedies these problems. AFS-BM achieves
this by joint optimization for simultaneous feature selection and model
training. In particular, we do the joint optimization and binary masking to
continuously adapt the set of features and model parameters during the training
process. This approach leads to significant improvements in model accuracy and
a reduction in computational requirements. We provide an extensive set of
experiments where we compare AFS-BM with the established feature selection
methods using well-known datasets from real-life competitions. Our results show
that AFS-BM makes significant improvement in terms of accuracy and requires
significantly less computational complexity. This is due to AFS-BM's ability to
dynamically adjust to the changing importance of features during the training
process, which an important contribution to the field. We openly share our code
for the replicability of our results and to facilitate further research.
