  Nowadays, feature selection is frequently used in machine learning when there
is a risk of performance degradation due to overfitting or when computational
resources are limited. During the feature selection process, the subset of
features that are most relevant and least redundant is chosen. In recent years,
it has become clear that, in addition to relevance and redundancy, features'
complementarity must be considered. Informally, if the features are weak
predictors of the target variable separately and strong predictors when
combined, then they are complementary. It is demonstrated in this paper that
the synergistic effect of complementary features mutually amplifying each other
in the construction of two-tier decision trees can be interfered with by
another feature, resulting in a decrease in performance. It is demonstrated
using cross-validation on both synthetic and real datasets, regression and
classification, that removing or eliminating the interfering feature can
improve performance by up to 24 times. It has also been discovered that the
lesser the domain is learned, the greater the increase in performance. More
formally, it is demonstrated that there is a statistically significant negative
rank correlation between performance on the dataset prior to the elimination of
the interfering feature and performance growth after the elimination of the
interfering feature. It is concluded that this broadens the scope of feature
selection methods for cases where data and computational resources are
sufficient.
