  As the field of automated machine learning (AutoML) advances, it becomes
increasingly important to incorporate domain knowledge into these systems. We
present an approach for doing so by harnessing the power of large language
models (LLMs). Specifically, we introduce Context-Aware Automated Feature
Engineering (CAAFE), a feature engineering method for tabular datasets that
utilizes an LLM to iteratively generate additional semantically meaningful
features for tabular datasets based on the description of the dataset. The
method produces both Python code for creating new features and explanations for
the utility of the generated features. Despite being methodologically simple,
CAAFE improves performance on 11 out of 14 datasets -- boosting mean ROC AUC
performance from 0.798 to 0.822 across all dataset - similar to the improvement
achieved by using a random forest instead of logistic regression on our
datasets. Furthermore, CAAFE is interpretable by providing a textual
explanation for each generated feature. CAAFE paves the way for more extensive
semi-automation in data science tasks and emphasizes the significance of
context-aware solutions that can extend the scope of AutoML systems to semantic
AutoML. We release our $\href{https://github.com/automl/CAAFE}{code}$, a simple
$\href{https://colab.research.google.com/drive/1mCA8xOAJZ4MaB_alZvyARTMjhl6RZf0a}{demo}$
and a $\href{https://pypi.org/project/caafe/}{python\ package}$.
