  This study introduces a systematic framework to compare the efficacy of Large
Language Models (LLMs) for fine-tuning across various cheminformatics tasks.
Employing a uniform training methodology, we assessed three well-known
models-RoBERTa, BART, and LLaMA-on their ability to predict molecular
properties using the Simplified Molecular Input Line Entry System (SMILES) as a
universal molecular representation format. Our comparative analysis involved
pre-training 18 configurations of these models, with varying parameter sizes
and dataset scales, followed by fine-tuning them on six benchmarking tasks from
DeepChem. We maintained consistent training environments across models to
ensure reliable comparisons. This approach allowed us to assess the influence
of model type, size, and training dataset size on model performance.
Specifically, we found that LLaMA-based models generally offered the lowest
validation loss, suggesting their superior adaptability across tasks and
scales. However, we observed that absolute validation loss is not a definitive
indicator of model performance - contradicts previous research - at least for
fine-tuning tasks: instead, model size plays a crucial role. Through rigorous
replication and validation, involving multiple training and fine-tuning cycles,
our study not only delineates the strengths and limitations of each model type
but also provides a robust methodology for selecting the most suitable LLM for
specific cheminformatics applications. This research underscores the importance
of considering model architecture and dataset characteristics in deploying AI
for molecular property prediction, paving the way for more informed and
effective utilization of AI in drug discovery and related fields.
