  In this paper, we introduce GUIDE, a novel dataset tailored for the
advancement of Multimodal Large Language Model (MLLM) applications,
particularly focusing on Robotic Process Automation (RPA) use cases. Our
dataset encompasses diverse data from various websites including
Apollo(62.67\%), Gmail(3.43\%), Calendar(10.98\%) and Canva(22.92\%). Each data
entry includes an image, a task description, the last action taken, CoT and the
next action to be performed along with grounding information of where the
action needs to be executed. The data is collected using our in-house advanced
annotation tool NEXTAG (Next Action Grounding and Annotation Tool). The data is
adapted for multiple OS, browsers and display types. It is collected by
multiple annotators to capture the variation of design and the way person uses
a website.
  Through this dataset, we aim to facilitate research and development in the
realm of LLMs for graphical user interfaces, particularly in tasks related to
RPA. The dataset's multi-platform nature and coverage of diverse websites
enable the exploration of cross-interface capabilities in automation tasks. We
believe that our dataset will serve as a valuable resource for advancing the
capabilities of multi-platform LLMs in practical applications, fostering
innovation in the field of automation and natural language understanding. Using
GUIDE, we build V-Zen, the first RPA model to automate multiple websites using
our in-House Automation tool AUTONODE
