  We develop a new approach for feature selection via gain penalization in
tree-based models. First, we show that previous methods do not perform
sufficient regularization and often exhibit sub-optimal out-of-sample
performance, especially when correlated features are present. Instead, we
develop a new gain penalization idea that exhibits a general local-global
regularization for tree-based models. The new method allows for more
flexibility in the choice of feature-specific importance weights. We validate
our method on both simulated and real data and implement itas an extension of
the popular R package ranger.
