  In this paper, we highlight the critical issues of robustness and safety
associated with integrating large language models (LLMs) and vision-language
models (VLMs) into robotics applications. Recent works focus on using LLMs and
VLMs to improve the performance of robotics tasks, such as manipulation and
navigation. Despite these improvements, analyzing the safety of such systems
remains underexplored yet extremely critical. LLMs and VLMs are highly
susceptible to adversarial inputs, prompting a significant inquiry into the
safety of robotic systems. This concern is important because robotics operate
in the physical world where erroneous actions can result in severe
consequences. This paper explores this issue thoroughly, presenting a
mathematical formulation of potential attacks on LLM/VLM-based robotic systems
and offering experimental evidence of the safety challenges. Our empirical
findings highlight a significant vulnerability: simple modifications to the
input can drastically reduce system effectiveness. Specifically, our results
demonstrate an average performance deterioration of 19.4% under minor input
prompt modifications and a more alarming 29.1% under slight perceptual changes.
These findings underscore the urgent need for robust countermeasures to ensure
the safe and reliable deployment of advanced LLM/VLM-based robotic systems.
