  We study building multi-task agents in open-world environments. Without human
demonstrations, learning to accomplish long-horizon tasks in a large open-world
environment with reinforcement learning (RL) is extremely inefficient. To
tackle this challenge, we convert the multi-task learning problem into learning
basic skills and planning over the skills. Using the popular open-world game
Minecraft as the testbed, we propose three types of fine-grained basic skills,
and use RL with intrinsic rewards to acquire skills. A novel Finding-skill that
performs exploration to find diverse items provides better initialization for
other skills, improving the sample efficiency for skill learning. In skill
planning, we leverage the prior knowledge in Large Language Models to find the
relationships between skills and build a skill graph. When the agent is solving
a task, our skill search algorithm walks on the skill graph and generates the
proper skill plans for the agent. In experiments, our method accomplishes 40
diverse Minecraft tasks, where many tasks require sequentially executing for
more than 10 skills. Our method outperforms baselines by a large margin and is
the most sample-efficient demonstration-free RL method to solve Minecraft Tech
Tree tasks. The project's website and code can be found at
https://sites.google.com/view/plan4mc.
