  We consider the use of automated supervised learning systems for data tables
that not only contain numeric/categorical columns, but one or more text fields
as well. Here we assemble 18 multimodal data tables that each contain some text
fields and stem from a real business application. Our publicly-available
benchmark enables researchers to comprehensively evaluate their own methods for
supervised learning with numeric, categorical, and text features. To ensure
that any single modeling strategy which performs well over all 18 datasets will
serve as a practical foundation for multimodal text/tabular AutoML, the diverse
datasets in our benchmark vary greatly in: sample size, problem types (a mix of
classification and regression tasks), number of features (with the number of
text columns ranging from 1 to 28 between datasets), as well as how the
predictive signal is decomposed between text vs. numeric/categorical features
(and predictive interactions thereof). Over this benchmark, we evaluate various
straightforward pipelines to model such data, including standard two-stage
approaches where NLP is used to featurize the text such that AutoML for tabular
data can then be applied. Compared with human data science teams, the fully
automated methodology that performed best on our benchmark (stack ensembling a
multimodal Transformer with various tree models) also manages to rank 1st place
when fit to the raw text/tabular data in two MachineHack prediction
competitions and 2nd place (out of 2380 teams) in Kaggle's Mercari Price
Suggestion Challenge.
