  Automated Machine Learning (AutoML) offers a promising approach to streamline
the training of machine learning models. However, existing AutoML frameworks
are often limited to unimodal scenarios and require extensive manual
configuration. Recent advancements in Large Language Models (LLMs) have
showcased their exceptional abilities in reasoning, interaction, and code
generation, presenting an opportunity to develop a more automated and
user-friendly framework. To this end, we introduce AutoM3L, an innovative
Automated Multimodal Machine Learning framework that leverages LLMs as
controllers to automatically construct multimodal training pipelines. AutoM3L
comprehends data modalities and selects appropriate models based on user
requirements, providing automation and interactivity. By eliminating the need
for manual feature engineering and hyperparameter optimization, our framework
simplifies user engagement and enables customization through directives,
addressing the limitations of previous rule-based AutoML approaches. We
evaluate the performance of AutoM3L on six diverse multimodal datasets spanning
classification, regression, and retrieval tasks, as well as a comprehensive set
of unimodal datasets. The results demonstrate that AutoM3L achieves competitive
or superior performance compared to traditional rule-based AutoML methods.
Furthermore, a user study highlights the user-friendliness and usability of our
framework, compared to the rule-based AutoML methods.
