  Large Language Models (LLMs) stand at the forefront of a number of Natural
Language Processing (NLP) tasks. Despite the widespread adoption of LLMs in
NLP, much of their potential in broader fields remains largely unexplored, and
significant limitations persist in their design and implementation. Notably,
LLMs struggle with structured data, such as graphs, and often falter when
tasked with answering domain-specific questions requiring deep expertise, such
as those in biology and chemistry. In this paper, we explore a fundamental
question: Can LLMs effectively handle molecule prediction tasks? Rather than
pursuing top-tier performance, our goal is to assess how LLMs can contribute to
diverse molecule tasks. We identify several classification and regression
prediction tasks across six standard molecule datasets. Subsequently, we
carefully design a set of prompts to query LLMs on these tasks and compare
their performance with existing Machine Learning (ML) models, which include
text-based models and those specifically designed for analysing the geometric
structure of molecules. Our investigation reveals several key insights:
Firstly, LLMs generally lag behind ML models in achieving competitive
performance on molecule tasks, particularly when compared to models adept at
capturing the geometric structure of molecules, highlighting the constrained
ability of LLMs to comprehend graph data. Secondly, LLMs show promise in
enhancing the performance of ML models when used collaboratively. Lastly, we
engage in a discourse regarding the challenges and promising avenues to harness
LLMs for molecule prediction tasks. The code and models are available at
https://github.com/zhiqiangzhongddu/LLMaMol.
