  Categorical variables often appear in datasets for classification and
regression tasks, and they need to be encoded into numerical values before
training. Since many encoders have been developed and can significantly impact
performance, choosing the appropriate encoder for a task becomes a
time-consuming yet important practical issue. This study broadly classifies
machine learning models into three categories: 1) ATI models that implicitly
perform affine transformations on inputs, such as multi-layer perceptron neural
network; 2) Tree-based models that are based on decision trees, such as random
forest; and 3) the rest, such as kNN. Theoretically, we prove that the one-hot
encoder is the best choice for ATI models in the sense that it can mimic any
other encoders by learning suitable weights from the data. We also explain why
the target encoder and its variants are the most suitable encoders for
tree-based models. This study conducted comprehensive computational experiments
to evaluate 14 encoders, including one-hot and target encoders, along with
eight common machine-learning models on 28 datasets. The computational results
agree with our theoretical analysis. The findings in this study shed light on
how to select the suitable encoder for data scientists in fields such as fraud
detection, disease diagnosis, etc.
