  In this study, we leverage LLM to enhance the semantic analysis and develop
similarity metrics for texts, addressing the limitations of traditional
unsupervised NLP metrics like ROUGE and BLEU. We develop a framework where LLMs
such as GPT-4 are employed for zero-shot text identification and label
generation for radiology reports, where the labels are then used as
measurements for text similarity. By testing the proposed framework on the
MIMIC data, we find that GPT-4 generated labels can significantly improve the
semantic similarity assessment, with scores more closely aligned with clinical
ground truth than traditional NLP metrics. Our work demonstrates the
possibility of conducting semantic analysis of the text data using
semi-quantitative reasoning results by the LLMs for highly specialized domains.
While the framework is implemented for radiology report similarity analysis,
its concept can be extended to other specialized domains as well.
