  Intelligence is key to advancing integrated circuit (IC) fabrication. Recent
breakthroughs in Large Multimodal Models (LMMs) have unlocked unparalleled
abilities in understanding images and text, fostering intelligent fabrication.
Leveraging the power of LMMs, we introduce FabGPT, a customized IC fabrication
large multimodal model for wafer defect knowledge query. FabGPT manifests
expertise in conducting defect detection in Scanning Electron Microscope (SEM)
images, performing root cause analysis, and providing expert question-answering
(Q&A) on fabrication processes. FabGPT matches enhanced multimodal features to
automatically detect minute defects under complex wafer backgrounds and reduce
the subjectivity of manual threshold settings. Besides, the proposed modulation
module and interactive corpus training strategy embed wafer defect knowledge
into the pre-trained model, effectively balancing Q&A queries related to defect
knowledge and original knowledge and mitigating the modality bias issues.
Experiments on in-house fab data (SEM-WaD) show that our FabGPT achieves
significant performance improvement in wafer defect detection and knowledge
querying.
