  This study compares the design practices and performance of ChatGPT 4.0, a
large language model (LLM), against graduate engineering students in a 48-hour
prototyping hackathon, based on a dataset comprising more than 100 prototypes.
The LLM participated by instructing two participants who executed its
instructions and provided objective feedback, generated ideas autonomously and
made all design decisions without human intervention. The LLM exhibited similar
prototyping practices to human participants and finished second among six
teams, successfully designing and providing building instructions for
functional prototypes. The LLM's concept generation capabilities were
particularly strong. However, the LLM prematurely abandoned promising concepts
when facing minor difficulties, added unnecessary complexity to designs, and
experienced design fixation. Communication between the LLM and participants was
challenging due to vague or unclear descriptions, and the LLM had difficulty
maintaining continuity and relevance in answers. Based on these findings, six
recommendations for implementing an LLM like ChatGPT in the design process are
proposed, including leveraging it for ideation, ensuring human oversight for
key decisions, implementing iterative feedback loops, prompting it to consider
alternatives, and assigning specific and manageable tasks at a subsystem level.
