  Hyper-parameters (HPs) are an important part of machine learning (ML) model
development and can greatly influence performance. This paper studies their
behavior for three algorithms: Extreme Gradient Boosting (XGB), Random Forest
(RF), and Feedforward Neural Network (FFNN) with structured data. Our empirical
investigation examines the qualitative behavior of model performance as the HPs
vary, quantifies the importance of each HP for different ML algorithms, and
stability of the performance near the optimal region. Based on the findings, we
propose a set of guidelines for efficient HP tuning by reducing the search
space.
