  In recent developments within the research community, the integration of
Large Language Models (LLMs) in creating fully autonomous agents has garnered
significant interest. Despite this, LLM-based agents frequently demonstrate
notable shortcomings in adjusting to dynamic environments and fully grasping
human needs. In this work, we introduce the problem of LLM-based human-agent
collaboration for complex task-solving, exploring their synergistic potential.
In addition, we propose a Reinforcement Learning-based Human-Agent
Collaboration method, ReHAC. This approach includes a policy model designed to
determine the most opportune stages for human intervention within the
task-solving process. We construct a human-agent collaboration dataset to train
this policy model in an offline reinforcement learning environment. Our
validation tests confirm the model's effectiveness. The results demonstrate
that the synergistic efforts of humans and LLM-based agents significantly
improve performance in complex tasks, primarily through well-planned, limited
human intervention. Datasets and code are available at:
https://github.com/XueyangFeng/ReHAC.
