  Lymph node metastasis (LNM) is a crucial factor in determining the initial
treatment for patients with lung cancer, yet accurate preoperative diagnosis of
LNM remains challenging. Recently, large language models (LLMs) have garnered
significant attention due to their remarkable text generation capabilities.
Leveraging the extensive medical knowledge learned from vast corpora, LLMs can
estimate probabilities for clinical problems, though their performance has
historically been inferior to data-driven machine learning models. In this
paper, we propose a novel ensemble method that combines the medical knowledge
acquired by LLMs with the latent patterns identified by machine learning models
to enhance LNM prediction performance. Initially, we developed machine learning
models using patient data. We then designed a prompt template to integrate the
patient data with the predicted probability from the machine learning model.
Subsequently, we instructed GPT-4o, the most advanced LLM developed by OpenAI,
to estimate the likelihood of LNM based on patient data and then adjust the
estimate using the machine learning output. Finally, we collected three outputs
from the GPT-4o using the same prompt and ensembled these results as the final
prediction. Using the proposed method, our models achieved an AUC value of
0.778 and an AP value of 0.426 for LNM prediction, significantly improving
predictive performance compared to baseline machine learning models. The
experimental results indicate that GPT-4o can effectively leverage its medical
knowledge and the probabilities predicted by machine learning models to achieve
more accurate LNM predictions. These findings demonstrate that LLMs can perform
well in clinical risk prediction tasks, offering a new paradigm for integrating
medical knowledge and patient data in clinical predictions.
