  Learning in environments with large state and action spaces, and sparse
rewards, can hinder a Reinforcement Learning (RL) agent's learning through
trial-and-error. For instance, following natural language instructions on the
Web (such as booking a flight ticket) leads to RL settings where input
vocabulary and number of actionable elements on a page can grow very large.
Even though recent approaches improve the success rate on relatively simple
environments with the help of human demonstrations to guide the exploration,
they still fail in environments where the set of possible instructions can
reach millions. We approach the aforementioned problems from a different
perspective and propose guided RL approaches that can generate unbounded amount
of experience for an agent to learn from. Instead of learning from a
complicated instruction with a large vocabulary, we decompose it into multiple
sub-instructions and schedule a curriculum in which an agent is tasked with a
gradually increasing subset of these relatively easier sub-instructions. In
addition, when the expert demonstrations are not available, we propose a novel
meta-learning framework that generates new instruction following tasks and
trains the agent more effectively. We train DQN, deep reinforcement learning
agent, with Q-value function approximated with a novel QWeb neural network
architecture on these smaller, synthetic instructions. We evaluate the ability
of our agent to generalize to new instructions on World of Bits benchmark, on
forms with up to 100 elements, supporting 14 million possible instructions. The
QWeb agent outperforms the baseline without using any human demonstration
achieving 100% success rate on several difficult environments.
