  This paper presents our team's participation in the MEDIQA-ClinicalNLP2024
shared task B. We present a novel approach to diagnosing clinical dermatology
cases by integrating large multimodal models, specifically leveraging the
capabilities of GPT-4V under a retriever and a re-ranker framework. Our
investigation reveals that GPT-4V, when used as a retrieval agent, can
accurately retrieve the correct skin condition 85% of the time using
dermatological images and brief patient histories. Additionally, we empirically
show that Naive Chain-of-Thought (CoT) works well for retrieval while Medical
Guidelines Grounded CoT is required for accurate dermatological diagnosis.
Further, we introduce a Multi-Agent Conversation (MAC) framework and show its
superior performance and potential over the best CoT strategy. The experiments
suggest that using naive CoT for retrieval and multi-agent conversation for
critique-based diagnosis, GPT-4V can lead to an early and accurate diagnosis of
dermatological conditions. The implications of this work extend to improving
diagnostic workflows, supporting dermatological education, and enhancing
patient care by providing a scalable, accessible, and accurate diagnostic tool.
