  Completing complex tasks in unpredictable settings like home kitchens
challenges robotic systems. These challenges include interpreting high-level
human commands, such as "make me a hot beverage" and performing actions like
pouring a precise amount of water into a moving mug. To address these
challenges, we present a novel framework that combines Large Language Models
(LLMs), a curated Knowledge Base, and Integrated Force and Visual Feedback
(IFVF). Our approach interprets abstract instructions, performs long-horizon
tasks, and handles various uncertainties. It utilises GPT-4 to analyse the
user's query and surroundings, then generates code that accesses a curated
database of functions during execution. It translates abstract instructions
into actionable steps. Each step involves generating custom code by employing
retrieval-augmented generalisation to pull IFVF-relevant examples from the
Knowledge Base. IFVF allows the robot to respond to noise and disturbances
during execution. We use coffee making and plate decoration to demonstrate our
approach, including components ranging from pouring to drawer opening, each
benefiting from distinct feedback types and methods. This novel advancement
marks significant progress toward a scalable, efficient robotic framework for
completing complex tasks in uncertain environments. Our findings are
illustrated in an accompanying video and supported by an open-source GitHub
repository (released upon paper acceptance).
