  Learning effective representations from raw data is crucial for the success
of deep learning methods. However, in the tabular domain, practitioners often
prefer augmenting raw column features over using learned representations, as
conventional tree-based algorithms frequently outperform competing approaches.
As a result, feature engineering methods that automatically generate candidate
features have been widely used. While these approaches are often effective,
there remains ambiguity in defining the space over which to search for
candidate features. Moreover, they often rely solely on validation scores to
select good features, neglecting valuable feedback from past experiments that
could inform the planning of future experiments. To address the shortcomings,
we propose a new tabular learning framework based on large language models
(LLMs), coined Optimizing Column feature generator with decision Tree reasoning
(OCTree). Our key idea is to leverage LLMs' reasoning capabilities to find good
feature generation rules without manually specifying the search space and
provide language-based reasoning information highlighting past experiments as
feedback for iterative rule improvements. Here, we choose a decision tree as
reasoning as it can be interpreted in natural language, effectively conveying
knowledge of past experiments (i.e., the prediction models trained with the
generated features) to the LLM. Our empirical results demonstrate that this
simple framework consistently enhances the performance of various prediction
models across diverse tabular benchmarks, outperforming competing automatic
feature engineering methods.
