  Engineering design is undergoing a transformative shift with the advent of
AI, marking a new era in how we approach product, system, and service planning.
Large language models have demonstrated impressive capabilities in enabling
this shift. Yet, with text as their only input modality, they cannot leverage
the large body of visual artifacts that engineers have used for centuries and
are accustomed to. This gap is addressed with the release of multimodal
vision-language models (VLMs), such as GPT-4V, enabling AI to impact many more
types of tasks. Our work presents a comprehensive evaluation of VLMs across a
spectrum of engineering design tasks, categorized into four main areas:
Conceptual Design, System-Level and Detailed Design, Manufacturing and
Inspection, and Engineering Education Tasks. Specifically in this paper, we
assess the capabilities of two VLMs, GPT-4V and LLaVA 1.6 34B, in design tasks
such as sketch similarity analysis, CAD generation, topology optimization,
manufacturability assessment, and engineering textbook problems. Through this
structured evaluation, we not only explore VLMs' proficiency in handling
complex design challenges but also identify their limitations in complex
engineering design applications. Our research establishes a foundation for
future assessments of vision language models. It also contributes a set of
benchmark testing datasets, with more than 1000 queries, for ongoing
advancements and applications in this field.
