  We introduce a novel framework of LLM agents named AGILE (AGent that
Interacts and Learns from Environments) designed to perform complex
conversational tasks with users, leveraging LLMs, memory, tools, and
interactions with experts. The agent's abilities include not only conversation
but also reflection, utilization of tools, and consultation with experts. We
formulate the construction of such an LLM agent as a reinforcement learning
problem, in which the LLM serves as the policy model. We fine-tune the LLM
using labeled data of actions and the PPO algorithm. We focus on question
answering and release a dataset for agents called ProductQA, comprising
challenging questions in online shopping. Our extensive experiments on
ProductQA and MedMCQA show that AGILE agents based on 13B and 7B LLMs trained
with PPO can outperform GPT-4 agents. Our ablation study highlights the
indispensability of memory, tools, consultation, reflection, and reinforcement
learning in achieving the agent's strong performance.
