  Cyber-physical systems (CPSs) integrate cyber and physical components and
enable them to interact with each other to meet user needs. The needs for CPSs
span rich application domains such as healthcare and medicine, smart home,
smart building, etc. This indicates that CPSs are all about solving real-world
problems. With the increasing abundance of sensing devices and effectors, the
problems wanted to solve with CPSs are becoming more and more complex. It is
also becoming increasingly difficult to extract and express CPS requirements
accurately. Problem frame approach aims to shape real-world problems by
capturing the characteristics and interconnections of components, where the
problem diagram is central to expressing the requirements. CPSs requirements
are generally presented in domain-specific documents that are normally
expressed in natural language. There is currently no effective way to extract
problem diagrams from natural language documents. CPSs requirements extraction
and modeling are generally done manually, which is time-consuming,
labor-intensive, and error-prone. Large language models (LLMs) have shown
excellent performance in natural language understanding. It can be interesting
to explore the abilities of LLMs to understand domain-specific documents and
identify modeling elements, which this paper is working on. To achieve this
goal, we first formulate two tasks (i.e., entity recognition and interaction
extraction) and propose a benchmark called CPSBench. Based on this benchmark,
extensive experiments are conducted to evaluate the abilities and limitations
of seven advanced LLMs. We find some interesting insights. Finally, we
establish a taxonomy of LLMs hallucinations in CPSs requirements modeling using
problem diagrams. These results will inspire research on the use of LLMs for
automated CPSs requirements modeling.
