  Robotic Process Automation (RPA) systems face challenges in handling complex
processes and diverse screen layouts that require advanced human-like
decision-making capabilities. These systems typically rely on pixel-level
encoding through drag-and-drop or automation frameworks such as Selenium to
create navigation workflows, rather than visual understanding of screen
elements. In this context, we present SmartFlow, an AI-based RPA system that
uses pre-trained large language models (LLMs) coupled with deep-learning based
image understanding. Our system can adapt to new scenarios, including changes
in the user interface and variations in input data, without the need for human
intervention. SmartFlow uses computer vision and natural language processing to
perceive visible elements on the graphical user interface (GUI) and convert
them into a textual representation. This information is then utilized by LLMs
to generate a sequence of actions that are executed by a scripting engine to
complete an assigned task. To assess the effectiveness of SmartFlow, we have
developed a dataset that includes a set of generic enterprise applications with
diverse layouts, which we are releasing for research use. Our evaluations on
this dataset demonstrate that SmartFlow exhibits robustness across different
layouts and applications. SmartFlow can automate a wide range of business
processes such as form filling, customer service, invoice processing, and
back-office operations. SmartFlow can thus assist organizations in enhancing
productivity by automating an even larger fraction of screen-based workflows.
The demo-video and dataset are available at
https://smartflow-4c5a0a.webflow.io/.
