  Embodied dialogue instruction following requires an agent to complete a
complex sequence of tasks from a natural language exchange. The recent
introduction of benchmarks (Padmakumar et al., 2022) raises the question of how
best to train and evaluate models for this multi-turn, multi-agent,
long-horizon task. This paper contributes to that conversation, by arguing that
imitation learning (IL) and related low-level metrics are actually misleading
and do not align with the goals of embodied dialogue research and may hinder
progress. We provide empirical comparisons of metrics, analysis of three
models, and make suggestions for how the field might best progress. First, we
observe that models trained with IL take spurious actions during evaluation.
Second, we find that existing models fail to ground query utterances, which are
essential for task completion. Third, we argue evaluation should focus on
higher-level semantic goals.
