  Machine learning algorithms often contain many hyperparameters (HPs) whose
values affect the predictive performance of the induced models in intricate
ways. Due to the high number of possibilities for these HP configurations and
their complex interactions, it is common to use optimization techniques to find
settings that lead to high predictive performance. However, insights into
efficiently exploring this vast space of configurations and dealing with the
trade-off between predictive and runtime performance remain challenging.
Furthermore, there are cases where the default HPs fit the suitable
configuration. Additionally, for many reasons, including model validation and
attendance to new legislation, there is an increasing interest in interpretable
models, such as those created by the Decision Tree (DT) induction algorithms.
This paper provides a comprehensive approach for investigating the effects of
hyperparameter tuning for the two DT induction algorithms most often used, CART
and C4.5. DT induction algorithms present high predictive performance and
interpretable classification models, though many HPs need to be adjusted.
Experiments were carried out with different tuning strategies to induce models
and to evaluate HPs' relevance using 94 classification datasets from OpenML.
The experimental results point out that different HP profiles for the tuning of
each algorithm provide statistically significant improvements in most of the
datasets for CART, but only in one-third for C4.5. Although different
algorithms may present different tuning scenarios, the tuning techniques
generally required few evaluations to find accurate solutions. Furthermore, the
best technique for all the algorithms was the IRACE. Finally, we found out that
tuning a specific small subset of HPs is a good alternative for achieving
optimal predictive performance.
