  We study the problem of developing autonomous agents that can follow human
instructions to infer and perform a sequence of actions to complete the
underlying task. Significant progress has been made in recent years, especially
for tasks with short horizons. However, when it comes to long-horizon tasks
with extended sequences of actions, an agent can easily ignore some
instructions or get stuck in the middle of the long instructions and eventually
fail the task. To address this challenge, we propose a model-agnostic
milestone-based task tracker (M-TRACK) to guide the agent and monitor its
progress. Specifically, we propose a milestone builder that tags the
instructions with navigation and interaction milestones which the agent needs
to complete step by step, and a milestone checker that systemically checks the
agent's progress in its current milestone and determines when to proceed to the
next. On the challenging ALFRED dataset, our M-TRACK leads to a notable 33% and
52% relative improvement in unseen success rate over two competitive base
models.
